Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium

ABSTRACT

An image processing apparatus includes an acquisition unit which acquires a parallax image generated based on a signal of a photoelectric converter among a plurality of photoelectric converters which receive light beams passing through partial pupil regions of an imaging optical system different from each other, and acquires a captured image generated by combining signals of the plurality of photoelectric converters, and an image processing unit which performs correction process so as to reduce a defect included in the parallax image based on the captured image.

This application is a Continuation of International Patent ApplicationNo. PCT/JP2016/002144, filed Apr. 21, 2016, which claims the benefit ofJapanese Patent Application No. 2015-095348, filed May 8, 2015 and No.2016-080328, filed Apr. 13, 2016, which are hereby incorporated byreference herein in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus which iscapable of correcting a parallax image.

Background Art

An image pickup apparatus has conventionally been known which is capableof dividing an exit pupil in an imaging lens into a plurality of pupilregions and simultaneously capturing a plurality of parallax imagescorresponding to the divided pupil regions. The captured parallax image(viewpoint image) is equivalent to LF (light field) data as informationon a spatial distribution and an angular distribution of a light amount.

PTL1 discloses an image pickup apparatus which uses a two-dimensionalimage pickup element that includes a single micro lens and a pluralityof divided photoelectric converters for one pixel. The dividedphotoelectric converters receive light beams passing through differentpartial pupil regions in an exit pupil in an imaging lens via the singlemicro lens, and thus the pupil is divided. A plurality of parallaximages corresponding to the divided partial pupil regions can begenerated based on light receiving signals of the respective dividedphotoelectric converters. PTL2 discloses an image pickup apparatus thatadds all of light receiving signals of divided photoelectric convertersto one another and generates a captured image.

However, a defect (flaw signal) such as a dot flaw and a line flaw,shading caused by the pupil division, a saturated signal etc. may occurin a part of the parallax image (viewpoint image) obtained by the imagepickup apparatuses disclosed in PTL1 and PTL2 and the parallax image maybe degraded.

CITATION LIST Patent Literature

PTL1 U.S. Pat. No. 4,410,804

PTL2 Japanese Patent Laid-Open No. 2001-083407

SUMMARY OF THE INVENTION

The present invention provides an image processing apparatus, an imagepickup apparatus, an image pickup method, and a non-transitorycomputer-readable storage medium, each of which can generate a parallaximage having an improved quality.

An image processing apparatus as one aspect of the present inventionincludes an acquisition unit configured to acquire a parallax imagegenerated based on a signal from one of a plurality of photoelectricconverters which receive light beams passing through partial pupilregions of an imaging optical system different from each other, and toacquire a captured image generated by combining a plurality of signalsfrom the plurality of photoelectric converters, and an image processingunit configured to correct the parallax image based on the capturedimage.

An image pickup apparatus as another aspect of the present inventionincludes an image pickup element including a plurality of arrayed pixelsthat include a plurality of photoelectric converters that receive lightbeams passing through partial pupil regions in an imaging optical systemdifferent from each other, an acquisition unit configured to acquire aparallax image generated based on a signal from one of the plurality ofphotoelectric converters, and to acquire a captured image generated bycombining signals from the plurality of photoelectric converters, and animage processing unit configured to correct the parallax image based onthe captured image.

An image processing method as another aspect of the present inventionincludes the steps of acquiring a parallax image generated based on asignal from one of a plurality of photoelectric converters which receivelight beams passing through partial pupil regions of an imaging opticalsystem different from each other, and acquiring a captured imagegenerated by combining a plurality of signals from the plurality ofphotoelectric converters, and correcting the parallax image based on thecaptured image.

A non-transitory computer-readable storage medium as another aspect ofthe present invention stores a program which causes a computer toexecute the steps of acquiring a parallax image generated based on asignal from one of a plurality of photoelectric converters which receivelight beams passing through partial pupil regions of an imaging opticalsystem different from each other, and acquiring a captured imagegenerated by combining a plurality of signals from the plurality ofphotoelectric converters, and correcting the parallax image based on thecaptured image.

Further features and aspects of the present invention will becomeapparent from the following description of exemplary embodiments withreference to the attached drawings.

The present invention can provide an image processing apparatus, animage pickup apparatus, an image pickup method, and a non-transitorycomputer-readable storage medium which are capable of generating aparallax image having an improved image quality.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image pickup apparatus in eachembodiment.

FIG. 2 is a diagram of illustrating a pixel array according to a firstembodiment.

FIG. 3A is a diagram of illustrating a pixel structure according to thefirst embodiment.

FIG. 3B is a diagram of illustrating a pixel structure according to thefirst embodiment.

FIG. 4 is an explanatory diagram of an image pickup element and a pupildividing function in each embodiment.

FIG. 5 is an explanatory diagram of the image pickup element and thepupil dividing function in each embodiment.

FIG. 6 is a diagram of a relationship between a defocus amount and animage shift amount in each embodiment.

FIG. 7A is an arrangement diagram of a parallax image in eachembodiment.

FIG. 7B is an arrangement diagram of the parallax image in eachembodiment.

FIG. 8 is an exemplary parallax image before a correction process isperformed in the first embodiment.

FIG. 9 is an exemplary parallax image after the correction process isperformed in the first embodiment.

FIG. 10 is another exemplary parallax image before the correctionprocess is performed in the first embodiment.

FIG. 11 is another exemplary parallax image after the correction processis performed in the first embodiment.

FIG. 12 is a diagram of illustrating a pixel array in a secondembodiment.

FIG. 13A is a diagram illustrating a pixel structure in the secondembodiment.

FIG. 13B is a diagram illustrating the pixel structure in the secondembodiment.

FIG. 14 is a schematic explanatory diagram of a refocus process in eachembodiment.

FIG. 15A is an explanatory view of a light amount distribution whenlight enters a micro lens formed on each pixel in a third embodiment.

FIG. 15B is an explanatory view of the light amount distribution whenthe light enters the micro lens formed on each pixel in the thirdembodiment.

FIG. 16 illustrates a light receiving rate distribution that depends ona light incident angle in the third embodiment.

FIG. 17 is a schematic view of a flow of a correction process in thethird embodiment.

FIG. 18 is a schematic view of a flow of the correction process in thethird embodiment.

FIG. 19A is an explanatory view of shading in the third embodiment.

FIG. 19B is an explanatory view of shading in the third embodiment.

FIG. 19C is an explanatory view of shading in the third embodiment.

FIG. 20A is an explanatory view of a projection signal of a capturedimage in the third embodiment.

FIG. 20B is an explanatory view of a projection signal of a firstviewpoint image in the third embodiment.

FIG. 20C is an explanatory view of a shading function in the thirdembodiment.

FIG. 21 illustrates an illustrative (demosaiced) captured image in thethird embodiment.

FIG. 22 is an illustrative first (demosaiced) viewpoint image beforeshading is corrected in the third embodiment.

FIG. 23 is an illustrative first viewpoint (first corrected) image (thathas been demosaiced) after shading is corrected in the third embodiment.

FIG. 24 is an illustrative first viewpoint (first corrected) image (thathas been demosaiced) after shading is corrected before a defect iscorrected in the third embodiment.

FIG. 25 is an illustrative first viewpoint (second corrected) image(that has been demosaiced) after shading is corrected and a defect iscorrected in the third embodiment.

FIG. 26 is an illustrative second viewpoint (first corrected) image(that has been demosaiced) before shading is corrected in the thirdembodiment.

FIG. 27 is an illustrative second corrected viewpoint image (that hasbeen demosaiced) after shading is corrected in the third embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to the accompanied drawings.

First Embodiment

Referring now to FIG. 1, a schematic configuration of an image pickupapparatus according to the first embodiment of the present inventionwill be described. FIG. 1 is a block diagram of an image pickupapparatus 100 (camera) in this embodiment. The image pickup apparatus100 is a digital camera system that includes a camera body and aninterchangeable lens (imaging optical system or image pickup opticalsystem) removably attached to the camera body. However, this embodimentis not limited to this example, and is applicable to an image pickupapparatus that includes a camera body and a lens which are integratedwith each other.

A first lens unit 101 is disposed at the forefront side (object side)among a plurality of lens units that constitute an imaging lens (imagingoptical system), and is held on a lens barrel so as to move back andforth in a direction of an optical axis OA (optical axis direction). Astop/shutter 102 (aperture stop) adjusts its opening diameter to controla light amount in capturing an image, and also functions as a shutter tocontrol an exposure time in capturing a still image. A second lens unit103 moves integrally with the stop/shutter 102 back and forth in theoptical axis direction, and has a zoom function for amagnification-varying operation in conjunction with the back-and-forthmotion of the first lens unit 101. A third lens unit 105 is a focus lensunit that moves back and forth in the optical axis direction forfocusing (a focus operation). An optical lowpass filter 106 is anoptical element that reduces a false color or a moire of a capturedimage (captured image).

An image pickup element 107 (image sensor) performs a photoelectricconversion of an object image (optical image) formed by the imagingoptical system, and for example, includes a CMOS sensor or a CCD sensor,and its peripheral circuit. As the image pickup element 107, forexample, a two-dimensional single plate color sensor is used whichincludes a primary color mosaic filter having a Bayer array formed onlight receiving pixels having m pixels in a horizontal direction andhaving n pixels in a vertical direction in an on-chip configuration.

A zoom actuator 111 rotationally moves (drives) an unillustrated camcylinder to move the first lens unit 101 and the second lens unit 103along the optical axis direction for the magnification-varyingoperation. The stop/shutter actuator 112 controls the opening diameterof the stop/shutter 102 to adjust the light amount (imaging lightamount) and also controls the exposure time in capturing the stillimage. A focus actuator 114 moves the third lens unit 105 in the opticalaxis direction for focusing.

An electronic flash 115 is an illumination device to be used forilluminating the object. The electronic flash 115 can use a flashillumination device that includes a xenon tube or an illumination devicethat includes an LED (light emitting diode) continuously emitting light.An AF auxiliary lighting unit 116 projects an image of a mask having apredetermined opening pattern onto the object via a projection lens.This configuration can improve a focus detection capability for a darkobject or an object with a low contrast.

A CPU 121 is a control apparatus (control unit or controller) thatgoverns various controls over the image pickup apparatus 100. The CPU121 includes a processor, a ROM, a RAM, an A/D converter, a D/Aconverter, a communication interface circuit, and the like. The CPU 121reads out and executes a predetermined program stored in the ROM todrive various circuits of the image pickup apparatus 100 and to performa series of operations such as focus detection (AF), imaging(photographing), image processing, or recording.

An electronic flash control circuit 122 controls lighting on and off ofthe electronic flash 115 in synchronization with the imaging operation.An auxiliary light driving circuit 123 controls lighting on and off ofthe AF auxiliary lighting unit 116 in synchronization with the focusdetection operation. An image pickup element driving circuit 124controls the imaging operation of the image pickup element 107 and alsoperforms the A/D conversion of the acquired image signal to send it tothe CPU 121.

An image processing circuit 125 (image processing apparatus) providesprocesses such as a γ (gamma) conversion, a color interpolation, or aJPEG (Joint Photographic Experts Group) compression on the image dataoutput from the image pickup element 107. In this embodiment, the imageprocessing circuit 125 includes an acquisition unit 125 a and an imageprocessing unit 125 b (corrector). The acquisition unit 125 a acquires acaptured image and at least one parallax image (viewpoint image) fromthe image pickup element 107. The captured image is an image generatedby combining a plurality of signals (first signal and second signal)from a plurality of photoelectric converters (first subpixel and secondsubpixel) which receive light beams passing through different partialpupil regions of the imaging optical system. The parallax image(viewpoint image) is an image generated based on a signal from aphotoelectric converter (first subpixel or second subpixel) among theplurality of photoelectric converters. The image processing unit 125 bperforms a correction process (corrects a defect) so as to reduce adefect included in the parallax image based on the captured image.

A focus driving circuit 126 (focus driver) drives the focus actuator 114based on the focus detection result to move the third lens unit 105along the optical axis direction for focusing. A stop/shutter drivingcircuit 128 drives the stop/shutter actuator 112 to control the openingdiameter of the stop/shutter 102. A zoom driving circuit 129 (zoomdriver) drives the zoom actuator 111 in response to a zoom operation bya user.

A display device 131 (display unit) includes for example, an LCD (liquidcrystal display). The display device 131 displays information on animaging mode of the image pickup apparatus 100, a pre-captured previewimage, a captured image for confirmation use, an in-focus statedisplaying image in the focus detection, or the like. An operatingmember 132 (operating switch unit) includes a power switch, a release(imaging trigger) switch, a zooming switch, an imaging mode selectionswitch, and the like. The release switch is a two-step switch in ahalf-pressed state (in a state where SW1 is ON) and in a fully pressedstate (in a state where SW2 is ON). A recording medium 133 is, forexample, a flash memory that is removable from the image pickupapparatus 100, and records a captured image (image data).

Referring now to FIGS. 2, 3A, and 3B, a pixel array and a pixelstructure of the image pickup element 107 according to this embodimentwill be described. FIG. 2 is a diagram illustrating the pixel array ofthe image pickup element 107. FIGS. 3A and 3B are diagrams illustratingthe pixel structure of the image pickup element 107, and FIG. 3Aillustrates a plan view of a pixel 200G of the image pickup element 107(view in a +z direction) and FIG. 3B illustrates an a-a sectional viewin FIG. 3A (viewed in a −z direction).

FIG. 2 illustrates the pixel array (array of imaging pixels) on theimage pickup element 107 (two-dimensional CMOS sensor) in a range of 4columns×4 rows. In this embodiment, each of the imaging pixels (pixels200R, 200G, and 200B) includes two subpixels 201 and 202. Therefore,FIG. 2 illustrates the array of the subpixels in a range of 8 columns×4rows.

As illustrated in FIG. 2, a pixel group 200 of 2 columns×2 rows includesthe pixels 200R, 200G, and 200B in a Bayer array. In the pixel group200, the pixel 200R having a spectral sensitivity for R (red) isdisposed at the upper left, the pixels 200G having a spectralsensitivity for G (green) are disposed at the upper right and at thelower left, and the pixel 200B having a spectral sensitivity for B(blue) is disposed at the lower right. Each of the pixels 200R, 200G,and 200B (each imaging pixel) includes the subpixels 201 and 202 arrayedin 2 columns×1 row. The subpixel 201 is a pixel which receives a lightbeam passing through a first pupil region in the imaging optical system.The subpixel 202 is a pixel which receives a light beam passing througha second pupil region in the imaging optical system.

As illustrated in FIG. 2, the image pickup element 107 includes manyimaging pixels of 4 columns×4 rows (subpixels of 8 columns×4 rows)arranged on its surface, and outputs an imaging signal (subpixelsignal). In the image pickup element 107 of this embodiment, a period Pof the pixels (imaging pixels) is 4 μm, and the number N of pixels(imaging pixels) is horizontally 5,575 columns×vertically 3725rows=approximately 20.75 million pixels. In the image pickup element107, a period P_(SUB) of the subpixels in a column direction is 2 μm,and the number N_(SUB) of the subpixels is horizontally 11,150columns×vertically 3,725 rows=approximately 41.50 million pixels.Alternatively, the image pickup element 107 may have a period P of thepixels of 6 μm and the number N of pixels (imaging pixels) ofhorizontally 6,000 columns×vertically 4,000 rows=approximately 24.00million pixels. Alternatively, in the image pickup element 107, a periodP_(SUB) of the subpixels in a column direction may be 3 μm, and thenumber N_(SUB) of the subpixels may be horizontally 12,000columns×vertically 4,000 rows=approximately 48.00 million pixels.

As illustrated in FIG. 3B, the pixel 200G of this embodiment is providedwith a micro lens 305 at a light receiving surface side of the pixel tocondense the incident light. The plurality of micro lenses 305 aretwo-dimensionally arrayed, and each of the micro lenses 305 is disposedat a position away from the light receiving surface by a predetermineddistance in a z-axis direction (direction of an optical axis directionOA). In the pixel 200G, a photoelectric converter 301 and aphotoelectric converter 302 (photoelectric converters) are formed bydividing the pixel into N_(H) (two divisions) in an x direction and intoN_(V) (one division) in a y direction. The photoelectric converter 301and the photoelectric converter 302 correspond to the subpixel 201 andthe subpixel 202, respectively.

Each of the photoelectric converters 301 and 302 is configured as aphotodiode having a p-i-n structure that includes a p-type layer and ann-type layer and an intrinsic layer between the p-type layer and then-type layer. If necessary, the intrinsic layer can be excluded and aphotodiode with a p-n junction may be applicable. The pixel 200G (eachpixel) is provided with a color filter 306 between the micro lens 305and each of the photoelectric converters 301 and 302. If necessary, aspectral transmittance of the color filter 306 can be changed for eachsubpixel, or alternatively the color filter may be excluded.

As illustrated in FIGS. 3A and 3B, the light entering the pixel 200G iscondensed by the micro lens 305 and is dispersed by the color filter306, and then the dispersed lights are received by the photoelectricconverters 301 and 302. In each of the photoelectric converters 301 and302, pairs of an electron and a hole are generated depending on a lightreceiving amount and they are separated in a depletion layer, and thenthe electrons with a negative charge are accumulated in the n-typelayer. On the other hand, the holes are excluded to the outside of theimage pickup element 107 through the p-type layer connected to aconstant voltage source (not illustrated). The electrons accumulated inthe n-type layers of the photoelectric converters 301 and 302 aretransferred to an electrostatic capacitance (FD) through a transfer gateto be converted to a voltage signal.

Referring now to FIG. 4, a pupil dividing function of the image pickupelement 107 will be described. FIG. 4 is an explanatory diagram of thepupil dividing function of the image pickup element 107, and illustratesthe pupil division in one pixel portion. FIG. 4 illustrates an a-asectional view of the pixel structure illustrated in FIG. 3A viewed fromthe +y direction and an exit pupil plane of the imaging optical system.In FIG. 4, in order to correspond to a coordinate axis of the exit pupilplane, the x-axis and y-axis in the sectional view are inverted withrespect to the x-axis and y-axis of FIGS. 3A and 3B.

In FIG. 4, a partial pupil region 501 (first partial pupil region) forthe subpixel 201 (first subpixel) has an approximately conjugaterelationship, via the micro lens 305, with the light receiving surfaceof the photoelectric converter 301 whose center of gravity is displaced(decentered) in the −x direction. Thus, the partial pupil region 501represents a pupil region which is capable of receiving light by thesubpixel 201. The center of gravity of the partial pupil region 501 forthe subpixel 201 is displaced (decentered) in the +x direction on apupil plane. A partial pupil region 502 (second partial pupil region)for the subpixel 202 (second subpixel) has an approximately conjugaterelationship, via the micro lens 305, with the light receiving surfaceof the photoelectric converter 302 whose center of gravity is displaced(decentered) in the +x direction. Thus, the partial pupil region 502represents a pupil region which is capable of receiving light by thesubpixel 202. The center of gravity of the partial pupil region 502 forthe subpixel 202 is displaced (decentered) in the −x direction on thepupil plane. A pupil region 500 can receive light over the entire regionof the pixel 200G when the photoelectric converters 301 and 302(subpixels 201 and 202) are entirely combined.

The incident light is condensed on a focus position by the micro lens305. However, due to an influence of diffraction caused by wave natureof light, a diameter of a light condensed spot cannot be made smallerthan a diffraction limit L and it has a finite size. While the lightreceiving surface of each of the photoelectric converters 301 and 302has a length of around 1 to 2 μm, the condensed spot of the micro lens305 is around 1 μm. Accordingly, the partial pupil regions 501 and 502in FIG. 4 which have a conjugate relationship with the light receivingsurface of the photoelectric converters 301 and 302 via the micro lens305 are not clearly divided due to a diffraction blur, and thus a lightreceiving rate distribution (pupil intensity distribution) is obtained.

FIG. 5 is an explanatory view of the image pickup element 107 and thepupil dividing function. The light beams passing through the differentpartial pupil regions 501 and 502 in the pupil region in the imagingoptical system enter each pixel of the image pickup element 107 on animaging plane 600 of the image pickup element 107 at angles differentfrom each other, and are received by the subpixels 201 and 202 dividedinto 2×1. This embodiment describes an illustrative pupil regionbisected in a horizontal direction, but the present invention is notlimited to this embodiment and the pupil may be divided in a verticaldirection, if necessary.

In this embodiment, the image pickup element 107 includes the pluralityof subpixels that share a single micro lens and that receive a pluralityof light beams passing through different regions (first partial pupilregion and second partial pupil region) in a pupil of the imagingoptical system (imaging lens). The image pickup element 107 includes, asthe plurality of subpixels, first subpixels (a plurality of subpixels201) and second subpixels (a plurality of subpixels 202).

In this embodiment, signals of the subpixels 201 and 202 are added(combined) to each other and read out for each pixel of the image pickupelement 107 so that a captured image is generated with a resolution ofthe number N of effective pixels. As described above, the captured imageis generated by combining light receiving signals of the plurality ofsubpixels (subpixels 201 and 202 in this embodiment) for each pixel.

In this embodiment, the light receiving signals of the plurality ofsubpixels 201 are collected to generate a first parallax image. Thefirst parallax image is subtracted from the captured image to generate asecond parallax image. However, this embodiment is not limited to thisexample, and the light receiving signals of the plurality of subpixels202 may be collected to generate the second parallax image. Thus, theparallax image is generated based on the light receiving signals of theplurality of subpixels for each of the partial pupil regions that aredifferent from each other.

In this embodiment, each of the first parallax image, the secondparallax image, and the captured image is an image in the Bayer array.If necessary, each of the first parallax image, the second parallaximage, and the captured image may be demosaiced.

Referring now to FIG. 6, a description will be given of a relationshipbetween a defocus amount of the first parallax image acquired from thesubpixels 201 and the second parallax image acquired from the subpixels202 in the image pickup element 107, and an image shift amount. FIG. 6illustrates the relationship between the defocus amount and the imageshift amount. FIG. 6 illustrates the image pickup element 107 disposedon the imaging plane 600, and similarly to FIGS. 4 and 5, the exit pupilof the imaging optical system divided into two partial pupil regions 501and 502.

A defocus amount d is defined such that a distance from an imagingposition of an object to the imaging plane 600 is |d|, a front focusstate has a negative sign (d<0) in which the imaging position is locatedcloser to an object side than the imaging plane 600, and a rear focusstate has a positive sign (d>0) in which the imaging position is locatedat a side opposite to the object relative to the imaging plane 600. Inan in-focus state in which the imaging position of the object is on theimaging plane 600 (in-focus position), the defocus amount d=0 issatisfied. FIG. 6 illustrates an object 601 in the in-focus state (d=0)and an object 602 in the front focus state (d<0). The front focus state(d<0) and the rear focus state (d>0) will be collectively referred to asa defocus state (|d|>0).

In the front focus state (d<0), the light beam that has passed throughthe partial pupil region 501 (or partial pupil region 502) in the lightbeams from the object 602 is condensed once. Thereafter, the light beamspreads to a width Γ1 (Γ2) around a center position of gravity G1 (G2)of the light beam, and a blurred image is formed on the imaging plane600. The blurred image is received by the subpixels 201 (subpixels 202)constituting each pixel arrayed in the image pickup element 107, and thefirst parallax image (second parallax image) is generated. Therefore,the first parallax image (second parallax image) is recorded as ablurred object image in which the object 602 is blurred with the widthΓ1 (Γ2) at the center position of gravity G1 (G2) on the imaging plane600. The blurred width Γ1 (Γ2) of the object image approximatelyincreases in proportion as the absolute value |d| of the defocus amountd increases. Similarly, an absolute value |p| of an image shift amount pof the object image between the first and second parallax images (i.e.which is equivalent to a difference of the center positions of gravityof the light beams (G1−G2)) approximately increases as the absolutevalue |d| of the defocus amount d increases. This is similarly appliedto the rear focus state (d>0), but an image shift direction of theobject image between the first and second parallax images is opposite tothat in the front focus state.

As described above, according to this embodiment, the absolute value ofthe image shift amount between the first and second parallax imagesincreases as the absolute value of the defocus amount increases betweenthe first and second parallax images or the imaging signals obtained byadding the first and second parallax images.

A description will now be given of the correction process of theparallax image in this embodiment. In this embodiment, the image pickupelement 107 can output the captured image and at least one parallaximage (at least one of the first parallax image and the second parallaximage). The image processing circuit 125 (acquisition unit 125 a)acquires the captured image and the parallax image that are output fromthe image pickup element 107. Then, the image processing circuit 125(image processing unit 125 b) corrects (revises) the parallax imagebased on the captured image. If necessary, the acquisition unit 125 amay store the acquired captured image and at least one acquired parallaximage in a memory such as the recording medium 133 and the memory 134 toacquire the captured image and parallax image which are stored in thememory.

Due to the influence caused by short-circuiting of the transfer gateaccording to a circuit configuration or a drive method of the imagepickup element 107, a flaw signal (defect signal) may occur in theparallax image (first parallax image or second parallax image) and adefect such as a dot flaw and a line flaw may be included in theparallax image even if the captured image is normal. If necessary,defect information such as dot flaw information and line flawinformation inspected in the mass production step or the like can bepreviously stored in the memory. In this case, the image processingcircuit 125 (image processing unit 125 b) performs the correctionprocess of the parallax image by using the stored defect information. Ifnecessary, the image processing circuit 125 (inspector) may inspect theparallax image on the real-time basis (i.e., while a user uses the imagepickup apparatus 100) and determine a defect such as a dot flaw and aline flaw.

Referring now to FIGS. 7A and 7B, the correction process of the parallaximage according to this embodiment will be described. FIG. 7A is anarray diagram of the parallax image (first parallax image) in the Bayerarray. FIG. 7B is an array diagram of the captured image in the Bayerarray. In FIGS. 7A and 7B, pixel values (pixel signals) of the firstparallax image and the captured image at a position (j, i) of an j-thpixel in a row direction and an i-th pixel in a column direction aredefined as A(j, i) and I(j, i), respectively.

If the first parallax image includes the defect (line flaw) in the j-throw and the captured image is normal in the j-th row, it is necessary tocorrect the j-th row in the first parallax image. In this embodiment,the image processing circuit 125 (image processing unit 125 b) correctsthe first parallax image (pixel value at a position to be corrected inthe first parallax image) based on the captured image. If necessary, thesecond parallax image can be corrected similarly.

In this embodiment, a correction value (correction signal) of the firstparallax image at a position (j, i) where the defect occurs, i.e.,position to be corrected (first position) is defined as Ac(j, i). Theimage processing unit 125 b calculates the correction value Ac(j, i)according to the following expression (1), and corrects the firstparallax image by using the calculated correction value Ac(j, i) as thepixel value A(j, i) of the first parallax image.

$\begin{matrix}{{{Ac}\left( {j,i} \right)} = {\frac{\max \left( {{\sum\limits_{k = {i - 1}}^{i + 1}\left\lbrack {{A\left( {{j - 1},k} \right)} + {A\left( {{j + 1},k} \right)}} \right\rbrack},{A\; 0}} \right)}{\max \left( {{\sum\limits_{k = {i - 1}}^{i + 1}\left\lbrack {{I\left( {{j - 1},k} \right)} + {I\left( {{j + 1},k} \right)}} \right\rbrack},{I\; 0}} \right)} \times {I\left( {j,i} \right)}}} & (1)\end{matrix}$

In the expression (1), parameters A0 and I0 are values used to stabilizea calculation value and to suppress an amplification of a noise if thepixel value A of the first parallax image and the pixel value I of thecaptured image have low luminances (low luminance signals).

As described above, in this embodiment, the image processing unit 125 bperforms the correction process for the parallax image based on thecaptured image, i.e., replaces the pixel value A(j,i) of the parallaximage at the position to be corrected with the correction value Ac(j,i).Specifically, the image processing unit 125 b determines the correctionvalue Ac(j, i) for the parallax image by using the pixel value I(j, i)of the captured image and pixel values I(j2, i2) and A(j2, i2) of thecaptured image and the parallax image at a position (j2, i2)≠(j, i) nearthe position to be corrected.

In the expression (1), specific values of the parameters A0 and I0 canbe set as appropriate. For example, if the pupil dividing number Np is2, the parameters A0 and I0 can be set to be A0=I0/Np. Values of theparameters A0 and I0 may be changed depending on an imaging conditionsuch as a position (j, i) to be corrected, an ISO speed, an F-number ofthe imaging optical system, and an exit pupil distance. The values ofthe parameters A0 and I0 may be set based on a pixel value A of thefirst parallax image or a pixel value I of the captured image near (atthe periphery of) the position to be corrected.

FIG. 8 illustrates an example of the first (demosaiced) parallax imagein an in-focus state before the correction process according to thisembodiment is performed. FIG. 9 illustrates an example of the first(demosaiced) parallax image in the in-focus state after the correctionprocess according to this embodiment is performed. Similarly, FIG. 10illustrates an example of the first (demosaiced) parallax image in adefocus state before the correction process is performed. FIG. 11illustrates an example of the first (demosaiced) parallax image in thedefocus state after the correction process is performed. It isunderstood that the defect of the parallax image is corrected by thecorrection process according to this embodiment in each of the in-focusstate and the defocus state.

Referring now to FIG. 14, a refocus process according to this embodimentwill be described. The refocus process is performed by the imageprocessing circuit 125 (image processing unit 125 b as a refocus unit)based on an instruction of the CPU 121. FIG. 14 is an explanatorydiagram of the refocus process in one-dimensional direction (columndirection or horizontal direction) by a first signal (light receivingsignal of the first subpixel that forms the first parallax image) and asecond signal (light receiving signal of the second subpixel that formsthe second parallax image) acquired by the image pickup element 107according to this embodiment. In FIG. 14, symbol i denotes an integer,and schematically symbols Ai and Bi denote the first signal and thesecond signal, respectively, of an i-th pixel disposed on the imagingplane 600 in the column direction in the image pickup element 107. Thefirst signal Ai is a light receiving signal output based on the lightbeam entering the i-th pixel at a principal ray angle θa (correspondingto the partial pupil region 501 in FIG. 5). The second signal Bi is alight receiving signal output based on the light beam entering the i-thpixel at a principal ray angle θb (corresponding to the partial pupilregion 502 in FIG. 5).

Each of the first signal Ai and the second signal Bi has incident angleinformation as well as light intensity distribution information.Therefore, the first signal Ai is moved parallel (translated) to avirtual imaging plane 610 at the angle θa and the second signal Bi ismoved parallel (translated) to the virtual imaging plane 610 at theangle θb, and then these signals are added to generate the refocussignal on the virtual imaging plane 610. The parallel movement of thefirst signal Ai to the virtual imaging plane 610 at the angle θacorresponds to a shift by +0.5 pixel in the column direction, and theparallel movement of the second signal Bi to the virtual imaging plane610 at the angle θb corresponds to a shift by −0.5 pixel in the columndirection. Accordingly, when the first signal Ai and the second signalBi relatively shifted by +1 pixel to add the first signal Ai to thecorresponding second signal (Bi+1) and to combine the first signal Aiwith the second signal (Bi+1), the refocus signal on the virtual imagingplane 610 can be generated. Similarly, when the first signal Ai and thesecond signal Bi are shifted by an integral multiple of the pixel pitch(i.e. integer shift is performed) and these signals are added to eachother, the shift addition signal (refocus signal) on each virtualimaging plane can be generated according to the integer shift amount.

In this embodiment, the influence of the defect included in at least oneof the parallax images (at least one of the first parallax image and thesecond parallax image) is removed or reduced by the correction process.Therefore, it is possible to perform the refocus process based on thecorrected parallax image. Accordingly, the refocus process by using eachof signals (first signal and second signal) that form the parallaximages can be performed with a high accuracy.

Second Embodiment

Referring now to FIG. 12 and FIGS. 13A and 13B, an image pickupapparatus according to a second embodiment of the present invention willbe described. This embodiment is different from the first embodiment inthat a captured image is generated based on the first to fourth parallaximages as a plurality of parallax images, instead of generating thecaptured image based on the first and second parallax images.

FIG. 12 illustrates the pixel array of the image pickup element 107according to this embodiment. FIGS. 13A and 13B are diagramsillustrating the pixel structure of the image pickup element 107, andFIGS. 13A and 13B are a plan view of a pixel 200G of the image pickupelement 107 (viewed from the +z direction) and a sectional view along aline a-a in FIG. 13A (viewed from the −z direction), respectively.

FIG. 12 illustrates the pixel array (array of imaging pixels) of theimage pickup element 107 (two-dimensional CMOS sensor) in a range of 4columns×4 rows. In this embodiment, each of the imaging pixels (pixels200R, 200G, and 200B) includes four subpixels 201, 202, 203, and 204.Therefore, FIG. 12 illustrates the array of the subpixels in a range of8 columns×8 rows.

As illustrated in FIG. 12, a pixel group 200 of 2 columns×2 rowsincludes the pixels 200R, 200G, and 200B in a Bayer array. In otherwords, in the pixel group 200, the pixel 200R having a spectralsensitivity for R (red) is disposed at the upper left, the pixels 200Ghaving a spectral sensitivity for G (green) are disposed at the upperright and at the lower left, and the pixel 200B having a spectralsensitivity for B (blue) is disposed at the lower right. Each of thepixels 200R, 200G, and 200B (each imaging pixel) includes the subpixels201, 202, 203, and 204 arrayed in 2 columns×2 row. The subpixel 201 is apixel which receives a light beam passing through a first pupil regionof the imaging optical system. The subpixel 202 is a pixel whichreceives a light beam passing through a second pupil region of theimaging optical system. The subpixel 203 is a pixel which receives alight beam passing through a third pupil region of the imaging opticalsystem. The subpixel 204 is a pixel which receives a light beam passingthrough a fourth pupil region in the imaging optical system.

As illustrated in FIG. 12, the image pickup element 107 includes manyimaging pixels of 4 columns×4 rows (subpixels of 8 columns×8 rows)arranged on a surface, and outputs an imaging signal (subpixel signal).In the image pickup element 107 of this embodiment, a period P of thepixels (imaging pixels) is 4 μm, and the number N of the pixels (imagingpixels) is horizontally 5,575 columns×vertically 3,725rows=approximately 20.75 million pixels. In the image pickup element107, a period P_(SUB) of the subpixels in a column direction is 2 μm,and the number N_(SUB) of the subpixels is horizontally 11,150columns×vertically 7,450 rows=approximately 83.00 million pixels.Alternatively, the image pickup element 107 may have a period P of thepixels (imaging pixels) of 6 μm and the number N of the pixels (imagingpixels) of horizontally 6,000 columns×vertically 4,000rows=approximately 24.00 million pixels. Alternatively, a period P_(SUB)of the subpixels in a column direction may be 3 μm, and the numberN_(SUB) of the subpixels may be horizontally 12,000 columns×vertically4,000 rows=approximately 48.00 million pixels.

As illustrated in FIG. 13B, the pixel 200G of this embodiment isprovided with a micro lens 305 at a light receiving surface side of thepixel to condense incident light. Each of the micro lenses 305 isdisposed at a position away from the light receiving surface by apredetermined distance in a z-axis direction (direction of an opticalaxis direction OA). In the pixel 200G, photoelectric converters 301,302, 303, and 304 (photoelectric converters) are formed by dividing thepixel into N_(H) (two divisions) in an x direction and into N_(V) (twodivision) in a y direction. The photoelectric converters 301 to 304correspond to the subpixels 201 to 204, respectively.

In this embodiment, the image pickup element 107 includes the pluralityof subpixels that share a single micro lens and that receive a pluralityof light beams passing through regions (first to fourth partial pupilregions) different from each other in a pupil of the imaging opticalsystem (imaging lens). The image pickup element 107 includes, as theplurality of subpixels, first subpixels (a plurality of subpixels 201),second subpixels (a plurality of subpixels 202), third subpixels (aplurality of subpixels 203), and fourth subpixels (a plurality ofsubpixels 204).

In this embodiment, signals of the subpixels 201, 202, 203, and 204 areadded (combined) and read out for each pixel of the image pickup element107 so that a captured image with a resolution of the effective pixelnumber N can be generated. As described above, the captured image isgenerated by combining light receiving signals of the plurality ofsubpixels (subpixels 201 to 204 in this embodiment) for each pixel.

In this embodiment, the light receiving signals of the plurality ofsubpixels 201 are collected to generate a first parallax image.Similarly, the light receiving signals of the plurality of subpixels 202are collected to generate a second parallax image, and the lightreceiving signals of the plurality of subpixels 203 are collected togenerate a third parallax image. Furthermore, in this embodiment, thefirst parallax image, the second parallax image, and the third parallaximage are subtracted from the captured image to generate a fourthparallax image. However, this embodiment is not limited to this example,and the light receiving signals of the plurality of subpixels 204 arecollected to generate the fourth parallax image. As described above, theparallax image is generated based on the light receiving signals of theplurality of subpixels for each of the partial pupil regions differentfrom each other.

In this embodiment, each of the captured image and the first to thirdparallax images (and the fourth parallax image) is an image in the Bayerarray. If necessary, each of the captured image and the first to thirdparallax images (and the fourth parallax image) may be demosaiced. Thecorrection process (defect correction) on the parallax image accordingto this embodiment is the same as that in the first embodiment, andaccordingly a descriptions thereof is omitted.

Third Embodiment

Next follows a description of the third embodiment according to thepresent invention. This embodiment is different from the firstembodiment in that the image processing unit 125 b performs a lightamount correction process (shading correction) of a parallax image basedon a captured image. In addition to the light amount correction processaccording to this embodiment, similar to the first embodiment, thecorrection process of the parallax image may be performed based on thecaptured image so as to reduce a defect contained in a parallax image.

A pupil region 500 illustrated in FIG. 4 has an approximately opticalconjugate relationship with a light receiving surface that includesphotoelectric converters 301 and 302 divided into 2×1 (firstphotoelectric converter to N_(LF)-th photoelectric converter dividedinto Nx×Ny) via a micro lens. The pupil region 500 is a pupil regionfrom which pixels 200G each including subpixels 201 and 202 (firstsubpixels to N_(LF)-th subpixel) can receive light.

FIGS. 15A and 15B are explanatory views of the light intensitydistribution where light enters the micro lens formed on each pixel.FIG. 15A illustrates a light intensity distribution of a sectionparallel to the optical axis of the micro lens. FIG. 15B illustrates alight intensity distribution of a section vertical to the optical axisof the micro lens. The incident light is condensed onto a focus positionby the micro lens. However, due to the influence of the diffraction bythe wave nature of the light, a diameter of a light condensed spotcannot be made smaller than a diffraction limit Δ and has a finite size.While the light receiving surface of the photoelectric converter has alength of about 1 to 2 μm, the condensed spot of the micro lens has alength of about 1 μm. Accordingly, due to the diffraction blurs, pupilpart areas 501 and 502 illustrated in FIG. 4 which have a conjugaterelationship with the light receiving surface of the photoelectricconverter via the micro lens are not pupil-divided and have a lightreceiving rate distribution (pupil intensity distribution) depending onthe light incident angle.

FIG. 16 is a view of a light receiving rate distribution (pupilintensity distribution) depending on the light incident angle. Theabscissa axis denotes a pupil coordinate, and the ordinate axis denotesa light receiving rate. A graph line L1 represented by a solid-line inFIG. 16 represents a pupil intensity distribution along the x axis inthe partial pupil region 501 (first partial area) in FIG. 4. The lightreceiving rate represented by the graph line L1 steeply rises from theleft end, reaches a peak, then gradually reduces with a smooth variationrate, and reaches the right end. The graph line L2 represented by abroken line in FIG. 16 represents a pupil intensity distribution alongthe x axis of the partial pupil region 502 (second partial pupilregion). Contrary to the graph line L1, the light receiving raterepresented by the graph line L2 steeply rises from the right end,reaches its peak, gradually decreases with a smooth variation rate, andreaches the left end. It is understood as illustrated in FIG. 16 thatthe smooth pupil division is performed.

As illustrated in FIG. 5, the photoelectric converters 301 and 302(first photoelectric converter to the N_(LF)-th photoelectric converter)correspond to the subpixels 201 and 202 (first subpixel to N_(LF)-thsubpixel). In each pixel on the image pickup element, the subpixels 201and 202 divided into 2×1 (first photoelectric converter to N_(LF)-thphotoelectric converter divided into Nx×Ny) receive light that haspassed different partial pupil regions on the partial pupil regions 501and 502 (first subpixel to N_(LF)-th subpixel). The LF data (input data(input image) representing the spatial distribution and the angulardistribution of the light intensity is obtained based on the signalreceived by each subpixel.

A captured image having a resolution of the pixel number N can begenerated based on the LF data (input image) for each pixel by combiningall the signals from the 2×1 divided subpixels 201 and 202 (Nx×Nydivided first photoelectric converter to N_(LF)-th photoelectricconverter) with one another.

A signal form a specific subpixel is selected for each pixel from the2×1 divided subpixels 201 and 202 (Nx×Ny divided first photoelectricconverter to N_(LF)-th photoelectric converter) based on the LF data(input image). Thereby, a viewpoint image corresponding to a specificpartial pupil region among the partial pupil regions 501 and 502 (firstsubpixel to N_(LF)-th subpixel) can be generated. For example, the firstviewpoint image (first parallax image) can be generated with aresolution of the pixel number N corresponding to the partial pupilregion 501 in the imaging optical system by selecting a signal from thesubpixel 201. This is true of another subpixel.

As discussed, the image pickup element according to this embodiment hasa plurality of arrayed pixels with a plurality of photoelectricconverters configured to receive light beams that have passed differentpartial pupil regions in the imaging optical system, and can acquire theLF data (input data). This embodiment performs the image processing(correction process), such as a flaw correction and a shadingcorrection, for the first viewpoint image and the second viewpoint image(first viewpoint image to N_(LF)-th viewpoint image) and generates anoutput image.

Referring now to FIGS. 17 and 18, a description will be given of animage processing method for generating an output image by performing acorrection process for the first viewpoint image and the secondviewpoint image (first viewpoint image to N_(LF)-th viewpoint image)based on the LF data (input image) acquired by the image pickup element107 and the captured image. FIGS. 17 and 18 are schematic views of aflow of the correction process according to this embodiment. The processin FIGS. 17 and 18 is mainly executed by the image processing circuit125 (acquisition unit 125 a and image processing unit 125 b) based on acommand of the CPU 121.

Initially, in a stage prior to the step S1 in FIG. 17 (or in theunillustrated step 0), the image processing circuit 125 (acquisitionunit 125 a) generates (acquires) a captured image and at least oneviewpoint image based on the LF data (input data) acquired by the imagepickup element 107. The captured image is an image generated inaccordance with a pupil region into which different partial pupilregions in the imaging optical system are combined. The viewpoint imageis an image generated for each different partial pupil region in theimaging optical system.

In the step S0, initially, the image processing circuit 125 inputs theLF data (input image) acquired by the image pickup element 107.Alternatively, the image processing circuit 125 may use the LF data(input image) previously captured by the image pickup element 107 andstored in the recording medium.

Next, in the step S0, the image processing circuit 125 generates acaptured image depending on the pupil region into which differentpartial pupil regions (first partial pupil region and the second partialpupil region) are combined in the imaging optical system. The LF data(input image) will be referred to as LF. In addition, a subpixel signalthat is the i_(s)-th (1≤i_(s)≤Nx) in the column direction and thej_(s)-th (1≤j_(s)≤Ny) in the row direction will be referred to as a k-thsubpixel signal in each pixel signal of the LF, wherek=Nx(j_(s)−1)+i_(s) (1≤k≤N_(LF)). The image processing circuit 125generates, as expressed in the following expression (2), a combinedimage as a captured image I(j, i) that is the i_(s)-th in the columndirection and the j_(s)-th in the row direction.

$\begin{matrix}{{I\left( {j,i} \right)} = {\sum\limits_{j_{S} = 1}^{N_{y}}{\sum\limits_{\;^{i_{S} = 1}}^{N_{x}}{{{LF}\left( {{{N_{y}\left( {j - 1} \right)} + j_{S}},{{N_{x}\left( {i - 1} \right)} + i_{s}}} \right)}.}}}} & (2)\end{matrix}$

In order to maintain the good S/N of the captured image (j, i), thisembodiment combines the subpixel signals of the expression (2) with eachother in an electrostatic capacitor (FD) in the image pickup elementbefore each subpixel signal is analog-to-digital-converted(A/D-converted). If necessary, before each subpixel signal isA/D-converted, this embodiment may combine the subpixel signals of theexpression (2) with each other in converting the electric chargesaccumulated in the electrostatic capacitor (FD) in the image pickupelement into the voltage signal. If necessary, after each subpixelsignal is A/D-converted, this embodiment may combine the subpixelsignals of the expression (2) with each other.

This embodiment bisects each pixel in the X direction such as Nx=2,Ny=1, and N_(LF)=2. All of the signals from the subpixels 201 and 202bisected in the X direction (Nx×Ny divided first subpixel to N_(LF)subpixel) are combined for each pixel based on the input image (LF data)corresponding to the illustrative pixel arrangement in FIG. 2. Thereby,a captured image as the RGB signal in the Bayer array can be generatedwith a resolution of the pixel number N (=horizontal pupil numberN_(H)×the vertical pixel number N_(V)). Since the correction process tothe viewpoint image according to this embodiment uses the captured imagefor the reference image of a correction reference, a shading (lightamount) correction process and a dot flaw correction process, etc. areperformed for the captured image I(j, i). If necessary, another processmay be performed.

Next, in the step S0, the image processing circuit 125 generates thek-th viewpoint image I_(k)(j, i) that is the i-th in the columndirection and the j-th in the row direction corresponding to the k-thpartial pupil region in the imaging optical system as expressed in thefollowing expression (3).

I _(k)(j,i)=I _(N) _(x) _((j) _(S) _(−1)+i) _(S) (j,i)=LF(N _(y)(j−1)+j_(S) ,N _(x)(i−1)+i _(S)).   (3)

This embodiment bisects each pixel in the X direction such as Nx=2,Ny=1, N_(LF)=2, and k=1. This embodiment selects a signal of thesubpixel 201 bisected in the x direction, for each pixel based on the LFdata (input image) corresponding to the pixel arrangement illustrated inFIG. 2. Then, this embodiment generates a first viewpoint image I₁(j, i)which is an RGB signal in the Bayer array with a resolution of the pixelnumber N (=horizontal pupil number N_(H)×the vertical pixel numberN_(V)) based on the partial pupil regions 501 and 502 (first partialpupil region to N_(LF)-th partial pupil region). If necessary, k=2 maybe selected, and the second viewpoint I₂(j, i) corresponding to thepartial pupil region 502 in the imaging optical system may be generated.

As described above, the image processing unit 125 b generates a capturedimage corresponding to a pupil region into which different partial pupilregions are combined, based on the input image acquired by the imagepickup element having a plurality of pixels with a plurality ofphotoelectric converters configured to receive light beams that havepassed different partial pupil regions in the imaging optical system. Inaddition, the image processing unit 125 b generates at least oneviewpoint image for each different partial pupil region.

This embodiment generates the captured image I(j, i) as the RGB signalin the Bayer array and the first viewpoint image I₁(j, i) as the RGBsignal in the Bayer array based on the LF data (input image) acquired bythe image pickup element 107, and stores them in the recording medium.In addition, this embodiment generates a second viewpoint image I₂(j, i)based on the captured image I(j, i) and the first viewpoint image I₁(j,i). This configuration can provide image processing similar to that forthe captured image acquired by the conventional image pickup element inwhich the photoelectric converter on each pixel is not divided for thecaptured image I(j, i) in this embodiment. If necessary, in order tomake equivalent the process to each viewpoint image, the first viewpointimage I₁(j, i) and the second viewpoint image I₂(j, i) may be generatedand stored in the recording medium.

Next, in the step S1 in FIG. 17, the image processing unit 125 bperforms a shading correction process (light amount correction process)for each of RGB in the first viewpoint image I₁ (k-th viewpoint imageI_(k)) based on the captured image I(j, i).

Referring now to FIGS. 19A to 19C, a description will be given of theshading caused by the pupil shift between the first viewpoint image andthe second viewpoint image (first viewpoint image to N_(LF)-th viewpointimage). FIGS. 19A to 19C are explanatory views of shading, andillustrate a relationship among the partial pupil region 501 throughwhich the photoelectric converter 301 receives light, the partial pupilregion 502 through which the photoelectric converter 302 receives light,and an exit pupil 400 in the imaging optical system, at the peripheralimage height in the image pickup element 107. Corresponding elements inFIG. 4 will be designated by the same reference numerals. Thephotoelectric converters 301 and 302 (first photoelectric converter toN_(LF)-th photoelectric converter) correspond to the subpixels 201 and202 (first subpixel to the N_(LF) subpixel).

FIG. 19A illustrates an exit pupil distance D1 in the imaging opticalsystem equal to a set pupil distance Ds in the image pickup element 107.In this case, the exit pupil 400 in the imaging optical system isapproximately uniformly divided by the partial pupil regions 501 and502. On the contrary, as illustrated in FIG. 19B, where the exit pupildistance D1 in the imaging optical system is shorter than the set pupildistance Ds in the image pickup element 107, the pupil shift occursbetween the exit pupil 400 and entrance pupil for the image pickupelement 107 at the peripheral image height of the image pickup element107. As a result, the exit pupil 400 is non-uniformly divided.Similarly, as illustrated in FIG. 19C, where the exit pupil distance C1in the imaging optical system is longer than the set pupil distance Dsin the image pickup element 107, the pupil shift occurs between the exitpupil 400 and entrance pupil in the image pickup element 107 at theperipheral image height of the image pickup element 107. As a result,the exit pupil 400 is non-uniformly divided. Along with the non-uniformpupil division at the peripheral image height, the first viewpoint imageand the second viewpoint image have non-uniform intensities and shadingoccurs for each of RGB (color) in which one of the first viewpoint imageand the second viewpoint image has a high intensity and the otherintensity is low.

In order to generate each viewpoint image with a good image quality, theimage processing unit 125 b according to this embodiment performs ashading correction (light amount correction) for each of RGB in thefirst viewpoint image I₁ (k-th viewpoint image I_(k)) by using thecaptured image I(j, i) as a base or reference image.

In the step S1 in FIG. 17, the image processing circuit 125 initiallydetects a valid pixel V₁(j, i) in which any one of the captured imageI(j, i) and the first viewpoint image I₁(j, i) is unsaturated andnon-defective (flawless). The effective pixel in which any one of thecaptured image I(j, i) and the first viewpoint image I₁(j, i) isunsaturated and non-defective satisfies V₁(j, i)=1. An ineffective pixelin which any one of the captured image I(j, i) and the first viewpointimage I₁(j, i) is saturated or defective satisfies V₁(j, i)=0.Similarly, in the shading (light quantity) correction for the k-thviewpoint image I_(k), the effective pixel in which both of the capturedimage I(j, i) and the k-th viewpoint image I_(k)(j, i) is unsaturatedand non-defective satisfies V_(k)(j, i)=1.

Assume integers j₂ (1≤j₂≤N_(V)/2) and i₂ (1≤i₂≤N_(H)/2). Assume that thecaptured image I in the Bayer array in FIG. 2 includes captured imagesRI, GrI, GbI, and BI for R, Gr, Gb, and B. The R captured image isexpressed as RI(2j₂−1, 2i₂−1)=I(2j₂−1, 2i₂−1) and the Gr captured imageis expressed as GrI(2j₂−1, 2i₂)=I(2j₂−1, 2i₂). The Gb captured image isexpressed as GbI(2j₂, 2i₂−1)=I(2j₂, 2i₂−1), and the B captured image isexpressed as BI(2j₂, 2i₂)=I(2j₂, 2i₂). Similarly, assume that the k-thcaptured image I_(k) illustrated in FIG. 2 includes captured imagesRI_(k), GrI_(k), GbI_(k), and BI_(k) for R, Gr, Gb, and B. The Rcaptured image is expressed as RI_(k)(2j₂−1, 2i₂−1)=I_(k)(2j₂−1, 2i₂−1)and the Gr captured image is expressed as GrI_(k)(2j₂−1,2i₂)=I_(k)(2i₂−1, 2i₂). The Gb captured image is expressed asGbI_(k)(2j₂, 2i₂−1)=I_(k)(2j₂, 2i₂−1) and the B captured image isexpressed as BI_(k)(2j₂, 2i₂)=I_(k)(2j₂, 2i₂).

In the step S1, next, the image processing unit 125 b performs aprojection process for the captured images RI(2j₂−1, 2i₂−1), GrI(2j₂−1,2i₂), GbI(2j₂, 2i₂−1), and BI(2j₂, 2i₂). More specifically, for thecaptured images RI(2j₂−1, 2i₂−1), GrI(2j₂−1, 2i₂), GbI(2j₂, 2i₂−1), andBI(2j₂, 2i₂), the projection process is performed in the direction (ydirection) orthogonal to the pupil dividing direction (x direction) withexpressions (4A) to (4D). Thereby, the projection signals RP(2i₂−1),GrP(2i₂), GbP(2i₂−1), and BP(2i₂) of the captured images are generated.The saturated signal value or the defective signal value does notcontain information used to correctly detect shading for each of RGB inthe captured image. Hence, a product of the captured image and theeffective pixel V_(k) is calculated, the projection process is performed(numerators on the upper stages in expressions (4A) to (4D)) byexcluding the saturated signal value and the defective signal value, anda normalization is performed with the effective pixel number used forthe projection process (denominators on the upper stages in expressions(4A) to (4D)). Where the effective pixel number used for the projectionprocess is 0, the projection signal of the captured signal is set to 0by the lower stage in expression (4A) to the lower stage in expression(4D). The projection signal of the captured image is also set to 0 wherethe projection signal of the captured image becomes a negative signaldue to the noise influences. Similarly, the projection process isperformed for the k-th viewpoint images RI_(k)(2j₂−1, 2i²⁻¹),GrI_(k)(2j₂−1, 2i₂), GbI_(k)(2j₂, 2i₂−1) and BI_(k)(2j₂, 2i₂) in thedirection (y direction) orthogonal to the pupil dividing direction (xdirection) with expressions (4E) to (4H). This configuration generatesprojection signals RP_(k)(2i₂−1), GrP_(k)(2i₂), GbP_(k)(2i₂−1), andBP_(k)(2i₂) of the k-th viewpoint image.

$\begin{matrix}{{{RP}\left( {{2i_{2}} - 1} \right)} = \left\{ \begin{matrix}{\frac{\begin{matrix}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{{{RI}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)} \times}} \\{V_{k}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)}\end{matrix}}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)}},} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)}} \neq 0},} \\{0,} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)}} = 0},}\end{matrix} \right.} & \left( {4A} \right) \\{{G_{r}{P\left( {2i_{2}} \right)}} = \left\{ \begin{matrix}{\frac{\begin{matrix}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{G_{r}{I\left( {{{2j_{2}} - 1},{2i_{2}}} \right)} \times}} \\{V_{k}\left( {{{2j_{2}} - 1},{2i_{2}}} \right)}\end{matrix}}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{2i_{2}}} \right)}},} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{2i_{2}}} \right)}} \neq 0},} \\{0,} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{2i_{2}}} \right)}} = 0},}\end{matrix} \right.} & \left( {4B} \right) \\{{G_{b}{P\left( {{2i_{2}} - 1} \right)}} = \left\{ \begin{matrix}{\frac{\begin{matrix}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{G_{b}{I\left( {{2j_{2}},{{2i_{2}} - 1}} \right)} \times}} \\{V_{k}\left( {{2j_{2}},{{2i_{2}} - 1}} \right)}\end{matrix}}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{{2i_{2}} - 1}} \right)}},} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{{2i_{2}} - 1}} \right)}} \neq 0},} \\{0,} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{{2i_{2}} - 1}} \right)}} = 0},}\end{matrix} \right.} & \left( {4C} \right) \\{\mspace{79mu} {{{BP}\left( {2i_{2}} \right)} = \left\{ \begin{matrix}{\frac{\begin{matrix}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{{{BI}\left( {{2j_{2}},{2i_{2}}} \right)} \times}} \\{V_{k}\left( {{2j_{2}},{2i_{2}}} \right)}\end{matrix}}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{2i_{2}}} \right)}},} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{2i_{2}}} \right)}} \neq 0},} \\{0,} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{2i_{2}}} \right)}} = 0},}\end{matrix} \right.}} & \left( {4D} \right) \\{{{RP}_{k}\left( {{2i_{2}} - 1} \right)} = \left\{ \begin{matrix}{\frac{\begin{matrix}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{{{RI}_{k}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)} \times}} \\{V_{k}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)}\end{matrix}}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)}},} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)}} \neq 0},} \\{0,} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)}} = 0},}\end{matrix} \right.} & \left( {4E} \right) \\{{G_{r}{P_{k}\left( {2i_{2}} \right)}} = \left\{ \begin{matrix}{\frac{\begin{matrix}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{G_{r}{I_{k}\left( {{{2j_{2}} - 1},{2i_{2}}} \right)} \times}} \\{V_{k}\left( {{{2j_{2}} - 1},{2i_{2}}} \right)}\end{matrix}}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{2i_{2}}} \right)}},} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{2i_{2}}} \right)}} \neq 0},} \\{0,} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{{2j_{2}} - 1},{2i_{2}}} \right)}} = 0},}\end{matrix} \right.} & \left( {4F} \right) \\{{G_{b}{P_{k}\left( {2i_{2}} \right)}} = \left\{ \begin{matrix}{\frac{\begin{matrix}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{G_{b}{I_{k}\left( {{2j_{2}},{{2i_{2}} - 1}} \right)} \times}} \\{V_{k}\left( {{2j_{2}},{{2i_{2}} - 1}} \right)}\end{matrix}}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{{2i_{2}} - 1}} \right)}},} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{{2i_{2}} - 1}} \right)}} \neq 0},} \\{0,} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{{2i_{2}} - 1}} \right)}} = 0},}\end{matrix} \right.} & \left( {4G} \right) \\{\mspace{79mu} {{{BP}_{k}\left( {2i_{2}} \right)} = \left\{ \begin{matrix}{\frac{\begin{matrix}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{{{BI}_{k}\left( {{2j_{2}},{2i_{2}}} \right)} \times}} \\{V_{k}\left( {{2j_{2}},{2i_{2}}} \right)}\end{matrix}}{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{2i_{2}}} \right)}},} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{2i_{2}}} \right)}} \neq 0},} \\{0,} & {{{\sum\limits_{j_{2} = 1}^{N_{V}/2}{V_{k}\left( {{2j_{2}},{2i_{2}}} \right)}} = 0},}\end{matrix} \right.}} & \left( {4H} \right)\end{matrix}$

A low-pass filter process follows the projection process of theexpressions (4A) to (4H). The low-pass filter process is performed forthe projection signals RP(2i₂−1), GrP(2i₂), GbP(2i₂−1), and BP(2i₂) ofthe captured image, and the projection signals RP_(k)(2i₂−1),GrP_(k)(2i₂), GbP_(k)(2i₂−1), and BP_(k)(2i₂) of the k-th viewpointimage. Thereby, the projection signal in the captured image is smoothed.Alternatively, the low-pass filter process may be omitted.

FIGS. 20A to 20C are explanatory views of the projection signals of thecaptured image, the projection signals of the first viewpoint image, andthe shading function. FIG. 20A illustrates the illustrative projectionsignals RP(R), GrP(G), GbP(G), and BP(B) of the captured image. FIG. 20Billustrates the illustrative projection signals RP₁(R), GrP₁(G),GbP₁(G), and BP₁(B) of the first viewpoint image. Each projection signalhas a plurality of undulations depending on the object. For a highlyaccurate shading correction of the first viewpoint image I₁(k-thviewpoint image I_(k)), it is necessary to separate the shadingcomponent of the first viewpoint image I₁(k-th viewpoint image I_(k))caused by the pupil shift for each of RGB from the signal component forthe RGB of the object.

In the step S1, the image processing unit 125 b calculates shadingsignals RS_(k)(2i₂−1), GrS_(k)(2i₂), GbS_(k)(2i₂−1), and BS_(k)(2i₂) foreach of RGB of the k-th viewpoint image I_(k) relative to the capturedimage as a reference by expressions (5A) to (5D).

$\begin{matrix}{{{RS}_{k}\left( {{2i_{2}} - 1} \right)} = \left\{ \begin{matrix}\frac{N_{LF} \times {{RP}_{k}\left( {{2i_{2}} - 1} \right)}}{{RP}\left( {{2i_{2}} - 1} \right)} & {{{{RP}\left( {{2i_{2}} - 1} \right)} > {{RP}_{k}\left( {{2i_{2}} - 1} \right)} > 0},} \\{0,} & {{otherwise},}\end{matrix} \right.} & \left( {5A} \right) \\{{G_{r}{S_{k}\left( {{2i_{2}} - 1} \right)}} = \left\{ \begin{matrix}\frac{N_{LF} \times G_{r}{P_{k}\left( {2i_{2}} \right)}}{G_{r}{P\left( {2i_{2}} \right)}} & {{{G_{r}{P\left( {2i_{2}} \right)}} > {G_{r}{P_{k}\left( {2i_{2}} \right)}} > 0},} \\{0,} & {{otherwise},}\end{matrix} \right.} & \left( {5B} \right) \\{{G_{b}{S_{k}\left( {{2i_{2}} - 1} \right)}} = \left\{ \begin{matrix}\frac{N_{LF} \times G_{b}{P_{k}\left( {{2i_{2}} - 1} \right)}}{G_{b}{P\left( {{2i_{2}} - 1} \right)}} & {{{G_{b}{P\left( {{2i_{2}} - 1} \right)}} > {G_{b}{P_{k}\left( {{2i_{2}} - 1} \right)}} > 0},} \\{0,} & {{otherwise},}\end{matrix} \right.} & \left( {5C} \right) \\{{{BS}_{k}\left( {{2i_{2}} - 1} \right)} = \left\{ \begin{matrix}\frac{N_{LF} \times {{BP}_{k}\left( {2i_{2}} \right)}}{{BP}\left( {2i_{2}} \right)} & {{{{BP}\left( {2i_{2}} \right)} > {{BP}_{k}\left( {2i_{2}} \right)} > 0},} \\{0,} & {{otherwise},}\end{matrix} \right.} & \left( {5D} \right)\end{matrix}$

It is necessary that a light receiving amount of the pixel is largerthan that of the subpixel, and the light receiving amount of thesubpixel is larger than 0 in calculating the shading component. Hence,where a conditional expression RP(2i₂−1)>RP_(k)(2i₂−1)>0 is satisfied, aratio is acquired by the expression (5A) between the R projection signalRP_(k)(2i₂−1) of the k-th viewpoint image and the R projection signalRP(2i₂−1) of the captured image. Then, the result is multiplied by thepupil dividing number N_(LF) for a normalization and the R shadingsignal RS_(k)(2i₂−1) of the k-th viewpoint image I_(k) is generated.Thereby, the R signal component of the object can be canceled out, andthe R shading component of the k-th viewpoint image I_(k) can beseparated. Where the conditional expression RP(2i₂−1)>RP_(k)(2i₂−1)>0 isnot satisfied, the R shading signal RS_(k)(2i₂−1) of the k-th viewpointimage I_(k) is set to 0.

Similarly, where a conditional expression GrP(2i₂)>GrP_(k)(2i₂)>0 issatisfied, a ratio is acquired by the expression (5B) between the Grprojection signal GrP_(k)(2i₂) of the k-th viewpoint image and the Grprojection signal GrP(2i₂) of the captured image. Then, the result ismultiplied by the pupil dividing number N_(LF) for a normalization andthe Gr shading signal GrS_(k)(2i₂) of the k-th viewpoint image I_(k) isgenerated. Thereby, the Gr signal component of the object can becanceled out, and the Gr shading component of the k-th viewpoint imageI_(k) can be separated. Where the conditional expressionGrP(2i₂)>GrP_(k)(2i₂)>0 is not satisfied, the Gr shading signalGrS_(k)(2i₂) of the k-th viewpoint image I_(k) is set to 0.

Similarly, where a conditional expression GbP(2i₂−1)>GbP_(k)(2i₂−1)>0 issatisfied, a ratio is acquired by the expression (5C) between the Gbprojection signal GbP_(k)(2i₂−1) of the k-th viewpoint image and the Gbprojection signal GbP(2i₂−1) of the captured image. Then, the result ismultiplied by the pupil dividing number N_(LF) for a normalization andthe Gb shading signal GbS_(k)(2i₂−1) of the k-th viewpoint image I_(k)is generated. Thereby, the Gb signal component of the object can becanceled out, and the Gb shading component of the k-th viewpoint imageI_(k) can be separated. Where the conditional expressionGbP(2i₂−1)>GbP_(k)(2i₂−1)>0 is not satisfied, the Gb shading signalGbS_(k)(2i₂−1) of the k-th viewpoint image I_(k) is set to 0.

Similarly, where a conditional expression BP(2i₂)>BP_(k)(2i₂)>0 issatisfied, a ratio is acquired by the expression (5D) between the Bprojection signal BP_(k)(2i₂) of the k-th viewpoint image and the Bprojection signal BP(2i₂) of the captured image. Then, the result ismultiplied by the pupil dividing number N_(LF) for a normalization andthe B shading signal BS_(k)(2i₂) of the k-th viewpoint image I_(k) isgenerated. Thereby, the B signal component of the object can becanceled, and the B shading component of the k-th viewpoint image I_(k)can be separated. Where the conditional expression BP(2i₂)>BP_(k)(2i₂)>0is not satisfied, the B shading signal BS_(k)(2i₂) of the k-th viewpointimage I_(k) is set to 0.

For a high shading correction accuracy, where an effectiveshading-signal number that satisfies RS_(k)(2i₂−1)>0, GrS_(k)(2i₂)>0,GbS_(k)(2i₂−1)>0, and BS_(k)(2i₂)>0 is equal to or higher than apredetermined number, the shading correction may be provided.

Next, in the step S1, the image processing unit 125 b performs acalculation process expressed by expressions (6A) to (6D). The shadingfunctions RSF_(k)(2i₂−1), GrSF_(k)(2i₂), GbSF_(k)(2i₂−1), andBSF_(k)(2i₂) of the k-th viewpoint image I_(k) for each of RGB are setto the smooth N_(SF)-th polynomial function to a positional variable inthe pupil dividing direction (x direction). The effective shading signalthat is generated by the expressions (5A) to (5D) and satisfiesRS_(k)(2i₂−1)>0, GrS_(k)(2i₂)>0, GbS_(k)(2i₂−1)>0, and BS_(k)(2i₂)>0 isset to a data point. Coefficients RSC_(k)(μ), GrSC_(k)(μ), GbSC_(k)(μ),and BSC_(k)(μ) in expressions (6A) to (6D) are calculated by these datapoints and parameter fitting with the least squares method.

$\begin{matrix}{{{{RSF}_{k}\left( {{2i_{2}} - 1} \right)} = {\sum\limits_{\mu = 0}^{N_{SF}}{{{RSC}_{k}(\mu)} \times \left( {{2i_{2}} - 1} \right)^{\mu}}}},} & \left( {6A} \right) \\{{{G_{r}{{SF}_{k}\left( {2i_{2}} \right)}} = {\sum\limits_{\mu = 0}^{N_{SF}}{G_{r}{{SC}_{k}(\mu)} \times \left( {2i_{2}} \right)^{\mu}}}},} & \left( {6B} \right) \\{{{G_{b}{{SF}_{k}\left( {{2i_{2}} - 1} \right)}} = {\sum\limits_{\mu = 0}^{N_{SF}}{G_{b}{{SC}_{k}(\mu)} \times \left( {{2i_{2}} - 1} \right)^{\mu}}}},} & \left( {6C} \right) \\{{{BSF}_{k}\left( {2i_{2}} \right)} = {\sum\limits_{\mu = 0}^{N_{SF}}{{{BSC}_{k}(\mu)} \times {\left( {2i_{2}} \right)^{\mu}.}}}} & \left( {6D} \right)\end{matrix}$

As described above, the shading functions RSF_(k)(2i₂−1), GrSF_(k)(2i₂),GbSF_(k)(2i₂−1), and BSF_(k)(2i₂) of the relative k-th viewpoint imageI_(k) for each of RGB on the basis of the captured image are generated.

FIG. 20C illustrates illustrative shading functions RSF₁(R), GrSF₁(G),GbSF₁(G), and BSF₁(B) for each of RGB of the first viewpoint image I₁relative to the captured image as a reference. The projection signal ofthe first viewpoint image in FIG. 20B and the projection signal of thecaptured image in FIG. 20A have undulations depending on the object. Onthe other hand, the undulation (the signal value of the object for eachof RGB) depending on the object can be canceled out by obtaining a ratiobetween the projection signal of the first viewpoint image and theprojection signal of the captured image, and the shading function of thesmooth first viewpoint image I₁ can be separated and generated for eachof RGB. While this embodiment uses the polynomial function for theshading function, the present invention is not limited to thisembodiment and may use a more general function depending on the shadingshape, if necessary.

Next, in the step S1 in FIG. 17, the image processing unit 125 b usesthe shading function for each of RGB with expressions (7A) to (7D), andperforms a shading (light quantity) correction process for the k-thviewpoint image I_(k)(j, i). Thereby, the k-th viewpoint (firstcorrected) image M₁I_(k)(j, i) after shading is corrected. The k-thviewpoint (first corrected) image M₁I_(k) in the Bayer array isexpressed for each of R, Gr, Gb, and B as follows. In other words, thek-th viewpoint (first corrected) image for R is set to RM₁I_(k)(2j₂−1,2i₂−1)=M₁I_(k)(2j₂−1, 2i₂−1), and the k-th viewpoint (first corrected)image for Gr is set to GrM₁I_(k)(2j₂−1, 2i₂)=M₁I_(k)(2j₂−1, 2i₂). Thek-th viewpoint (first corrected) image for Gb is set to GbM₁I_(k)(2j₂,2i₂−1)=M₁I_(k)(2j₂, 2i₂−1), and the k-th viewpoint (first corrected)image for B is set to BM₁I_(k)(2j₂, 2i₂)=M₁I_(k)(2j₂, 2i₂). Ifnecessary, the k-th viewpoint (first corrected) image M₁I_(k)(j, i)after shading is corrected may be set to an output image.

$\begin{matrix}{{{{RM}_{1}{I_{k}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)}} = \frac{{RI}_{k}\left( {{{2j_{2}} - 1},{{2i_{2}} - 1}} \right)}{{RSF}_{k}\left( {{2i_{2}} - 1} \right)}},} & \left( {7A} \right) \\{{{G_{r}M_{1}{I_{k}\left( {{{2j_{2}} - 1},{2i_{2}}} \right)}} = \frac{G_{r}{I_{k}\left( {{{2j_{2}} - 1},{2i_{2}}} \right)}}{G_{r}{{SF}_{k}\left( {2i_{2}} \right)}}},} & \left( {7B} \right) \\{{{G_{b}M_{1}{I_{k}\left( {{2j_{2}},{{2i_{2}} - 1}} \right)}} = \frac{G_{b}{I_{k}\left( {{2j_{2}},{{2i_{2}} - 1}} \right)}}{G_{b}{{SF}_{k}\left( {{2i_{2}} - 1} \right)}}},} & \left( {7C} \right) \\{{{{BM}_{1}{I_{k}\left( {{2j_{2}},{2i_{2}}} \right)}} = \frac{{BI}_{k}\left( {{2j_{2}},{2i_{2}}} \right)}{{BSF}_{k}\left( {{2i_{2}} - 1} \right)}},} & \left( {7D} \right)\end{matrix}$

Referring now to FIGS. 21 to 23, a description will be given of aneffect of the shading correction process (light amount correctionprocess) for each of RGB of the first viewpoint image I₁(j, i)illustrated in the step S1 in FIG. 17. FIG. 21 illustrates anillustrative (demosaiced) captured image I according to this embodiment.The illustrative captured image has a good image quality. FIG. 22illustrates an illustrative first (demosaiced) viewpoint image I₁ beforeshading is corrected in this embodiment. The shading occurs for each ofRGB due to the pupil shift between the exit pupil in the imaging opticalsystem and the entrance pupil of the image pickup element, and theluminance lowers and the RGB ratio is modulated on the right side of thefirst viewpoint image I₁(j, i). FIG. 23 illustrates an illustrativefirst viewpoint (first corrected) image M₁I₁ (post-demosaicing) aftershading is corrected in this embodiment. The luminance lowers and theRGB ratio is modulated due to the shading correction for each of RGBbased on the captured image, and the first viewpoint (first corrected)image M₁I₁(j, i) (post-demosaicing) with a good image quality isgenerated similar to the captured image.

This embodiment generates a captured image depending on the pupil regioninto which different partial pupil regions are combined, based on aninput image acquired by an image pickup element including a plurality ofpixels with a plurality of photoelectric converters configured toreceive light beams that have passed different partial pupil regions inthe imaging optical system. Then, this embodiment generates a pluralityof viewpoint images for each of the different partial pupil regions,performs image processing configured to correct the viewpoint imagebased on the captured image, and generates an output image. Thisembodiment performs image processing configured to correct a lightamount (shading) for each color or for each of RGB based on the capturedimage. This embodiment performs image processing configured to correct alight amount of a viewpoint image based on the projection signal of thecaptured image and the projection signal of the viewpoint image. Theconfiguration of this embodiment can provide a viewpoint image with agood quality.

Next, in the step S2 in FIG. 17, the image processing unit 125 bcorrects a defect of the k-th viewpoint (first corrected) image M₁I_(k)after shading is corrected, based on the captured image I. Thisembodiment illustrates an example of k=1, but the present invention isnot limited to this embodiment.

In this embodiment, due to the short-circuiting of the transfer gateetc., caused by the circuit configuration and driving method of theimage pickup element, only part of the k-th viewpoint image I_(k) (firstviewpoint image I₁) causes a defective signal or a dot or line defectalthough the captured image I is normal. If necessary, this embodimentmay previously store the dot defect information and line defectinformation inspected in the mass-production step etc. in the imageprocessing circuit 125 etc., and may perform the defect correctingprocess for the k-th viewpoint image I_(k) (first viewpoint image I₁)based on the recorded dot and line information. In addition, thisembodiment may inspect the k-th viewpoint image I_(k) (first viewpointimage I₁) on the real-time basis and provide a dot defect determinationor a line defect determination.

A description will now be given of the defect correction (step S2 inFIG. 17) according to this embodiment. This embodiment assumes that anodd row 2j_(D)−1 or an even row 2j_(D) in the k-th viewpoint image I_(k)is determined to have a line defect in the horizontal direction (xdirection) and an odd row 2j_(D)−1 or an even row 2j_(D) in the capturedimage I is not determined to have a line defect.

In the defect correction in the step S2 in this embodiment, the capturedimage I that is normal is set to a reference image and the defect of thek-th viewpoint (first corrected) image M₁I_(k) is corrected based on thecaptured image I. This embodiment corrects a defect by comparing asignal value of the k-th viewpoint (first correct) image M₁I_(k) at aposition that is not determined to be defective and a signal value ofthe captured image I at a position that is not determined to bedefective. In this comparison, it is necessary for a highly accuratedefect correction to remove the influence of the shading component foreach of RGB of the k-th viewpoint image I_(k) caused by the pupil shift,and to accurately compare the signal component for each of RGB of theobject between the k-th viewpoint image I_(k) and the captured image I.Hence, in the step S1, this embodiment previously corrects the shading(light amount) for each of RGB of the k-th viewpoint image, generatesthe k-th viewpoint (first corrected) image M₁I_(k), creates a shadingstate similar to that of the captured image I, and removes the influenceof a shading component. Thereafter, in the step S2, this embodimenthighly accurately corrects a defect in the k-th viewpoint (firstcorrected) image M₁I_(k) in which the shading is corrected, based on thecaptured image I.

In the step S2 in FIG. 17, this embodiment performs a defect correctionprocess for the signal that is determined to be defective, from part ofthe k-th viewpoint (first corrected) image M₁I_(k)(j, i) after theshading is corrected, based on the normal signal of the captured image Iand the normal signal of the k-th viewpoint (first corrected) imageM₁I_(k). Then, this embodiment generates the k-th viewpoint (secondcorrected) image M₂I_(k)(j, i) after the defect is corrected. Herein,the k-th viewpoint (second corrected) image M₂I_(k) in the Bayer arrayis expressed for each of R, Gr, Gb, and B as follows. In other words,the k-th viewpoint (second corrected) image for R is expressed asRM₂I_(k)(2j₂−1, 2i₂−1)=M₂I_(k)(2j₂−1, 2i₂−1), and the k-th viewpoint(second corrected) image for Gr is expressed as GrM₂I_(k)(2j₂−1,2i₂)=M₂I_(k)(2j₂−1, 2i₂). The k-th viewpoint (second corrected) imagefor Gb is expressed as GbM₂I_(k)(2j₂, 2i₂−1)=M₂I_(k)(2j₂, 2i₂−1), andthe k-th viewpoint (second corrected) image for B is expressed asBM₂I_(k)(2j₂, 2i₂)=M₂I_(k)(2j₂, 2i₂).

In the step S2, assume that a first position (2j_(D)−1, 2i_(D)−1) of thek-th viewpoint (first corrected) image M₁I_(k) for R is determined to bedefective. At this time, the defect correcting process is performed bythe following expression (8A) based on the captured image RI(2j_(D)−1,2i_(D)−1) at the first position, the k-th viewpoint (first corrected)image RM₁I_(k) at a second position for R that is not determined to bedefective, and the captured image RI at the second position. Thisconfiguration generates the k-th viewpoint (second corrected) imageRM₂I_(k)(2j_(D)−1, 2i_(D)−1) after the defect is corrected at the firstposition.

Assume that a first position (2j_(D)−1, 2i_(D)) of the k-th viewpoint(first corrected) image M₁I_(k) for Gr is determined to be defective. Atthis time, the defect correcting process is performed by the followingexpression (8B) based on the captured image GrI(2j_(D)−1, 2i_(D)) at thefirst position, the k-th viewpoint (first corrected) image GbM₁I_(k) atthe second position for Gb that is not determined to be defective, andthe captured image GbI at the second position. This configurationgenerates the k-th viewpoint (second corrected) imageGrM₂I_(k)(2j_(D)−1, 2i_(D)) after the defect is corrected at the firstposition.

Assume that a first position (2j_(D), 2i_(D)−1) of the k-th viewpoint(first corrected) image M₁I_(k) for Gb is determined to be defective. Atthis time, the defect correcting process is performed by the followingexpression (8C) based on the captured image GbI(2j_(D), 2i_(D)−1) at thefirst position, the k-th viewpoint (first corrected) image GrM₁I_(k) ata second position for Gr that is not determined to be defective, and thecaptured image GrI at the second position. This configuration generatesthe k-th viewpoint (second corrected) image GbM₂I_(k)(2j_(D), 2i_(D)−1)after the defect is corrected at the first position.

Assume that a first position (2j_(D), 2i_(D)) of the k-th viewpoint(first corrected) image M₁I_(k) for B is determined to be defective. Atthis time, the defect correcting process is performed by the followingexpression (8D) based on the captured image BI(2j_(D), 2i_(D)) at thefirst position, the k-th viewpoint (first corrected) image BM₁I_(k) at asecond position for B that is not determined to be defective, and thecaptured image BI at the second position. This configuration generatesthe k-th viewpoint (second corrected) image BM₂I_(k)(2j_(D), 2i_(D))after the defect is corrected at the first position.

$\begin{matrix}{{{{RM}_{2}{I_{k}\left( {{{2j_{D}} - 1},{{2i_{D}} - 1}} \right)}} = {{{RI}\left( {{{2j_{D}} - 1},{{2i_{D}} - 1}} \right)} \times \frac{\sum\limits_{\tau = {\pm 2}}{\sum\limits_{{\sigma = 0},{\pm 2}}{{RM}_{1}{I_{k}\left( {{{2j_{D}} - 1 + \tau},{{2i_{D}} - 1 + \sigma}} \right)}}}}{\sum\limits_{\tau = {\pm 2}}{\sum\limits_{{\sigma = 0},{\pm 2}}{{RI}\left( {{{2j_{D}} - 1 + \tau},{{2i_{D}} - 1 + \sigma}} \right)}}}}},} & \left( {8A} \right) \\{{{G_{r}M_{2}{I_{k}\left( {{{2j_{D}} - 1},{2i_{D}}} \right)}} = {G_{r}{I\left( {{{2j_{D}} - 1},{2i_{D}}} \right)} \times \frac{\sum\limits_{\tau = {\pm 1}}{\sum\limits_{{\sigma = 0},{\pm 1}}{G_{b}M_{1}{I_{k}\left( {{{2j_{D}} - 1 + \tau},{{2i_{D}} + \sigma}} \right)}}}}{\sum\limits_{\tau = {\pm 1}}{\sum\limits_{{\sigma = 0},{\pm 1}}{G_{b}{I\left( {{{2j_{D}} - 1 + \tau},{{2i_{D}} + \sigma}} \right)}}}}}},} & \left( {8B} \right) \\{{{G_{b}M_{2}{I_{k}\left( {{2j_{D}},{{2i_{D}} - 1}} \right)}} = {G_{b}{I\left( {{2j_{D}},{{2i_{D}} - 1}} \right)} \times \frac{\sum\limits_{\tau = {\pm 1}}{\sum\limits_{{\sigma = 0},{\pm 1}}{G_{r}M_{1}{I_{k}\left( {{{2j_{D}} + \tau},{{2i_{D}} - 1 + \sigma}} \right)}}}}{\sum\limits_{\tau = {\pm 1}}{\sum\limits_{{\sigma = 0},{\pm 1}}{G_{r}{I\left( {{{2j_{D}} + \tau},{{2i_{D}} - 1 + \sigma}} \right)}}}}}},} & \left( {8C} \right) \\{{{BM}_{2}{I_{k}\left( {{2j_{D}},{2i_{D}}} \right)}} = {{{BI}\left( {{2j_{D}},{2i_{D}}} \right)} \times {\frac{\sum\limits_{\tau = {\pm 2}}{\sum\limits_{{\sigma = 0},{\pm 2}}{{BM}_{1}{I_{k}\left( {{{2j_{D}} + \tau},{{2i_{D}} + \sigma}} \right)}}}}{\sum\limits_{\tau = {\pm 2}}{\sum\limits_{{\sigma = 0},{\pm 2}}{{BI}\left( {{{2j_{D}} - 1 + \tau},{{2i_{D}} + \sigma}} \right)}}}.}}} & \left( {8D} \right)\end{matrix}$

At most positions (j, i) that are not determined to be defective, thek-th viewpoint (second corrected) image M₂I_(k)(j, i) has the samesignal value as that of the k-th viewpoint (first corrected) imageM₁I_(k)(j, i) and M₂I_(k)(j, i)=M₁I_(k)(j, i) is established. Ifnecessary, the k-th viewpoint (second corrected) image M₂I_(k)(j, i)after the defect is corrected may be output as an output image.

Referring now to FIGS. 24 and 25, a description will be given of aneffect of the defect correction based on the normally captured image Iof the first viewpoint (first corrected) image M₁I₁ illustrated in thestep S2 in FIG. 17 in this embodiment. FIG. 24 illustrates an example ofthe first viewpoint (first corrected) image M₁I₁ (post-shadingcorrection and post-demosaicing) before the defect is corrected in thisembodiment. An illustrative line defect occurs in the horizontaldirection (x direction) at the center of the first viewpoint (firstcorrected) image M₁I₁(j, i). FIG. 25 illustrates an example of the firstviewpoint (second corrected) image M₂I₁ (post-shading correction andpost-demosaicing) after the defect is corrected in this embodiment. Dueto the defect correction based on the normally captured image I, theline defect in the horizontal direction (x direction) is corrected andthe first viewpoint (second corrected) image M₂I₁(j, i) is generatedwith a good quality after the defect is corrected, similar to thecaptured image.

This embodiment generates a captured image depending on a pupil regioninto which different partial pupil regions are combined, based on aninput image acquired by an image pickup element including a plurality ofpixels with a plurality of photoelectric converters configured toreceive light beams that have passed different partial pupil regions inthe imaging optical system. Then, this embodiment generates a pluralityof viewpoint images for each of the different partial pupil regions,performs image processing configured to correct a viewpoint image basedon the captured image, and generates an output image. This embodimentperforms image processing so as to correct and reduce the defectcontained in the viewpoint image based on the captured image. Thisembodiment performs image processing configured to correct a signalvalue of a viewpoint image at a first position that is determined to bedefective based on a signal value of a captured image at the firstposition. This embodiment performs signal processing configured tocorrect the signal value of the viewpoint image at the first positionbased on the signal value of the captured image at the first positionthat is determined to be defective, the signal value of the viewpointimage at the second position that is not determined to be defective, andthe signal value of the captured image at the second position.

In this embodiment, the image processing unit 125 b performs acorrection process (image processing) so as to reduce the defectcontained in the viewpoint image based on the captured image after thelight amount correction process of the viewpoint image is performedbased on the captured image. This configuration can generate theviewpoint image with a good quality.

Next, in the step S2 in FIG. 17, the image processing unit 125 bperforms a shading process for the k-th viewpoint (second corrected)image M₂I_(k)(j, i) after the defect is corrected, by using thefollowing expressions (9A) to (9D). Thereby, the k-th viewpoint (thirdcorrected) image M₃I_(k)(j, i) is generated.

RM ₃ I _(k)(2j ₂−1,2i ₂−1)=RSF _(k)(2i ₂−1)×RM ₂ I _(k)(2j ₂−1,2i ₂−1),  (9A)

G _(r) M ₃ I _(k)(2j ₂−1,2i ₂)=G _(r) SF _(k)(2i ₂)×G _(r) M ₂ I _(k)(2j₂−1,2i ₂),   (9B)

G _(b) M ₃ I _(k)(2j ₂,2i ₂−1)=G _(b) SF _(k)(2i ₂−1)×G _(b) M ₂ I_(k)(2j ₂,2i ₂−1),   (9C)

BM ₃ I _(k)(2j ₂,2i ₂)=BSF _(k)(2i ₂)×BM ₂ I _(k)(2j ₂,2i ₂).   (9D)

Now, the k-th viewpoint (third corrected) image M₃I_(k) in the Bayerarray is acquired for each of R, Gr, Gb, and B. The k-th viewpoint(third corrected) image for R is expressed as RM₃I_(k)(2j₂−1,2i₂−1)=M₃I_(k)(2j₂−1, 2i₂−1), and the k-th viewpoint (third corrected)image for Gr is expressed as GrM₃I_(k)(2j₂−1, 2i₂)=M₃I_(k)(2j₂−1, 2i₂).The k-th viewpoint (third corrected) image for Gb is expressed asGbM₃I_(k)(2j₂, 2i₂−1)=M₃I_(k)(2j₂, 2i₂−1), and the k-th viewpoint (thirdcorrected) image for B is expressed as BM₃I_(k)(2j₂, 2i₂)=M₃I_(k)(2j₂,2i₂).

In the step S3 in FIG. 18, a saturation signal process is performed forthe captured image I(j, i) and the k-th viewpoint (third corrected)image M₃I_(k)(j, i). This embodiment discusses an example where k=1 andN_(LF)=2.

In the step S3, initially, the saturation signal process is performedwith the following expression (10) for the captured image I(j, i) wherea maximum value of the captured signal is set to Imax, and the correctedcaptured image MI(j, i) is generated.

$\begin{matrix}{{{MI}\left( {j,i} \right)} = \left\{ \begin{matrix}{I_{\max},} & {{{I\left( {j,i} \right)} > I_{\max}},} \\{{I\left( {j,i} \right)},} & {{otherwise}.}\end{matrix} \right.} & (10)\end{matrix}$

Next, in the step S3, the image processing unit 125 b performs asaturation signal process corresponding to the shading state as in thefollowing expression (11), for the k-th viewpoint (third corrected)image M₃I_(k)(j, i) where the shading function for the Bayer array isset to SF_(k)(j, i). Thereby, the k-th viewpoint (fourth corrected)image M₄I_(k)(j, i) can be generated. Herein, the shading functionSF_(k)(j, i) for the Bayer array is calculated by the expression (6A) to(6D) based on shading functions RSF_(k)(2i₂−1), GrSF_(k)(2i₂),GbSF_(k)(2i₂−1), and BSF_(k)(2i₂) generated for each R, Gr, Gb, and B.In other words, assume that SF_(k)(2j₂−1, 2i₂−1)=RSF_(k)(2i₂−1),SF_(k)(2j₂−1, 2i₂)=GrSF_(k)(2i₂), SF_(k)(2j₂, 2i₂−1)=GbSF_(k)(2i₂−1),SF_(k)(2j₂, 2i₂)=BSF_(k)(2i₂).

$\begin{matrix}{{M_{4}{l_{k}\left( {j,i} \right)}} = \left\{ {\begin{matrix}{{{\frac{I_{\max}}{N_{LF}}S{F_{k}\left( {j,i} \right)}}\ ,}\ } & {{M_{3}{I_{k}\left( {j,i} \right)}} > {\frac{I_{\max}}{N_{LF}}{{SF}_{k}\left( {j,i} \right)}}} \\{{{M_{3}{I_{k}\left( {j,i} \right)}}\ ,}\ } & {otherwise}\end{matrix},} \right.} & (11)\end{matrix}$

In the step S4 in FIG. 18, the image processing unit 125 b generates thesecond viewpoint image I₂(j, i) based on expression (12), the correctedcaptured image MI(j, i), and the first viewpoint (fourth corrected)image M₄I₁(j, i).

I ₂(j,i)=MI(j,i)−M ₄ I ₁(j,i).   (12)

In this embodiment, the maximum signal value in the saturation of thefirst viewpoint (third corrected) image M₃I₁(j, i) may have the samemaximum signal value as the maximum signal value Imax in the saturationof the captured image I(j, i) due to the driving method of the imagepickup element 107 and the A/D conversion circuit configuration. In thiscase, when the second viewpoint image is generated by subtracting thefirst viewpoint (third corrected) image from the captured image as inthe expression (12) without the saturated signal process and the secondviewpoint image may have a saturated signal value, the saturated signalvalue may have an incorrect signal value of 0. Hence, in order toprevent this problem, the step S3 previously performs the saturationsignal process corresponding to the shading state for the captured imageI(j, i) and the k-th viewpoint (third corrected) image M₃I_(k)(j, i), soas to generate the corrected captured image MI(j, i) after thesaturation signal process is performed and the first viewpoint (fourthcorrected) image M₄I₁(j, i). Thereafter, the step S4 can generate thesecond viewpoint image I₂ corresponding to the correct saturated signalvalue using the expression (12) by generating the second viewpoint imageI₂(j, i).

In the step S5 in FIG. 18, the image processing unit 125 b performs theshading correction (light amount correction) for the first viewpoint(fourth corrected) image M₄I₁(j, i) and the second viewpoint image I₂(j,i). More specifically, the shading correction (light amount correction)is performed for the first viewpoint (fourth corrected) image M₄I₁(j, i)similar to the expressions (7A) and (7D) with the shading functionsRSF₁, GrSF₁, GbSF₁, and BSF₁ that have already been generated by theexpressions (6A) to (6D). Thereby, the first viewpoint (fifth corrected)image M₅I₁(j, i) is generated. Next, in the step S5, the imageprocessing unit 125 b performs the shading correction (light amountcorrection) for the second viewpoint image I₂(j, i) similar to theexpressions (4A) and (7D) based on the corrected captured image MI(j,i). Thereby, the second viewpoint (fifth corrected) image M₅I₂(j, i) isgenerated.

Finally, in the step S6 in FIG. 18, the image processing unit 125 bperforms the saturation signal process by the following expression (13)for the first viewpoint (fifth corrected) image M₅I₁(j, i) and thesecond viewpoint (fifth corrected) image M₅I₂(j, i). Thereby, thecorrected first viewpoint image MI₁(j, i) and the corrected secondviewpoint image MI₂(j, i) as the output images are generated.

$\begin{matrix}{{{MI}_{k}\left( {j,i} \right)} = \left\{ \begin{matrix}{{\frac{I_{\max}}{N_{LF}},}\mspace{7mu}} & {{{M_{5}{I_{k}\left( {j,i} \right)}} > \frac{I_{\max}}{N_{LF}}},} \\{{{M_{5}{I_{k}\left( {j,\ i} \right)}}\ ,}\ } & {{otherwise}.}\end{matrix} \right.} & (13)\end{matrix}$

Referring now to FIGS. 26 and 27, a description will be given of aneffect of the shading correction process (light amount correctionprocess) for each of RGB of the second viewpoint image I₂(j, i)illustrated in the step S5 in FIG. 17. FIG. 26 illustrates an example ofthe second viewpoint image I₂ (post-demosaicing) before the shading iscorrected in this embodiment. The pupil shift between the exit pupil inthe imaging optical system and the entrance pupil of the image pickupelement causes shading for each of RGB and thus the luminance reductionand the RGB ratio modulation on the left side of the second viewpointimage I₂(j, i). FIG. 27 illustrates an example of the corrected secondviewpoint image MI₂ (post-demosaicing) after the shading is corrected inthis embodiment. The shading correction for each of RGB based on thecaptured image corrects the luminance reduction and the RGB ratiomodulation, and generates the corrected second viewpoint image MI₂(j, i)with a good quality after the shading is corrected, similar to thecaptured image.

An image processing apparatus according to this embodiment is an imageprocessing apparatus having an image processing unit configured toperform the above image processing method. An image pickup apparatusaccording to this embodiment is an image pickup apparatus that includesan image pickup element including a plurality of arranged pixels with aplurality of subpixels configured to receive light beams that havepassed the different partial pupil regions in the imaging opticalsystem, and an image processing unit configured to perform the aboveimage processing method. The configuration of this embodiment cangenerate the viewpoint image with a good quality.

Fourth Embodiment

Next follows a description will be given of a fourth embodimentaccording to the present invention. This embodiment detects an imageshift amount distribution through a phase difference detection methodbased on the corrected first viewpoint image and the corrected secondviewpoint image (corrected first viewpoint image to corrected N_(LF)-thviewpoint image) generated in the third embodiment and a correlation(coincidence of the signal) between the corrected first viewpoint imageand the corrected second viewpoint image.

In generating the image shift amount distribution, initially, the colorcenters of gravity for RGB are accorded with one another for eachposition (j, i) based on the corrected k-th viewpoint image MIk (k=1 toN_(LF)) as the RGB signals in the Bayer array. A k-th viewpointluminance signal Yk is generated by the following expression (14).

$\begin{matrix}{{Y_{k}\left( {j,i} \right)} = {\begin{pmatrix}{{MI}_{k}\left( {{j - 1},{i - 1}} \right)} & {{MI}_{k}\left( {{j - 1},i} \right)} & {{MI}_{k}\left( {{j - 1},{i + 1}} \right)} \\{{MI}_{k}\left( {j,{i - 1}} \right)} & {{MI}_{k}\left( {j,i} \right)} & {{MI}_{k}\left( {j,{i + 1}} \right)} \\{{MI}_{k}\left( {{j + 1},{i - 1}} \right)} & {{MI}_{k}\left( {{j + 1},i} \right)} & {{MI}_{k}\left( {{j + 1},{i + 1}} \right)}\end{pmatrix}{\begin{pmatrix}\frac{1}{16} & \frac{2}{16} & \frac{1}{16} \\\frac{2}{16} & \frac{4}{16} & \frac{2}{16} \\\frac{1}{16} & \frac{2}{16} & \frac{1}{16}\end{pmatrix}.}}} & (14)\end{matrix}$

Next, in generating the image shift amount distribution, aone-dimensional bandpass filter process is performed for the firstviewpoint luminance signal Y₁ generated by the expression (14) in thepupil dividing direction (column direction) based on the corrected firstviewpoint image MI₁ as the RGB signal in the Bayer array, and generatesa first focus detecting signal dYA. In addition, this embodimentperforms a one-dimensional bandpass filter process for the secondviewpoint luminance signal Y₂ generated by the expression (14) in thepupil dividing direction (column direction) based on the correctedsecond viewpoint image MI₂, and generates a second focus detectingsignal dYB. The one-dimensional bandpass filter may use, for example, aprimary differential filter [1, 5, 8, 8, 8, 8, 5, 1, −1, −5, −8, −8, −8,−8, −5, −1] etc. If necessary, a passage band may be adjusted for theone-dimensional bandpass filter.

Next, in generating the image shift amount distribution, this embodimentshifts the first focus detection signal dYA and the second focusdetection signal dYB relative to each other in the pupil dividingdirection (column direction), calculates a correlation amountrepresenting the coincidence of the signal, and generates the imageshift amount distribution M_(DIS)(j, i) based on the correlation amount.This embodiment sets a first focus detection signal to dYA(j+j₂, i+i₂)that is the j₂-th (−n₂≤j₂≤n₂) in the row direction and the i₂-th(−m₂≤i₂≤m₂) in the column direction as the pupil dividing directionaround the position (j, i) as a center, and a second focus detectionsignal to dYB(j+j₂, i+i₂). Assume that a shift amount iss(−n_(s)≤s≤n_(s)). Then, a correlation amount COR_(EVEN)(j, i, s) ateach position (j, i) is calculated by the following expression (15A) anda correlation amount COR_(ODD)(j, i, s) is calculated by the followingexpression (15B).

$\begin{matrix}{{{COR}_{even}\left( {j,i,s} \right)} = {\sum\limits_{j_{2} = {- n_{2}}}^{n_{2}}{\sum\limits_{i_{2} = {- m_{2}}}^{m_{2}}{{{{dYA}\left( {{j + j_{2}},{i + i_{2} + s}} \right)} - {{dYB}\left( {{j + j_{2}},{i + i_{2} - s}} \right)}}}}}} & \left( {15A} \right) \\{{{COR}_{odd}\left( {j,i,s} \right)} = {\sum\limits_{j_{2} = {- n_{2}}}^{n_{2}}{\sum\limits_{i_{2} = {- m_{2}}}^{m_{2}}{{{{dYA}\left( {{j + j_{2}},{i + i_{2} + s}} \right)} - {{dYB}\left( {{j + j_{2}},{i + i_{2} - 1 - s}} \right)}}}}}} & \left( {15B} \right)\end{matrix}$

The correlation amount COR_(ODD)(j, i, s) is a correlation amount thatis made by shifting a shift amount between the first focus detectionsignal dYa and the second focus detection signal dYB by half a phase or−1 relative to the correlation amount COR_(EVEN)(j, i, s). Thisembodiment calculates and averages shift amounts of real number valuesthat minimizes the correlation amount, based on the correlation amountCOR_(EVEN)(j, i, s) and the correlation amount COR_(ODD)(j, i, s)through a subpixel calculation, and detects an image shift amountdistribution M_(DIS)(j, i).

In detecting an image shift amount by the phase difference method, thisembodiment evaluates the correlation amounts in the expressions (15A)and (15B), and detects the image shift amount based on the correlation(coincidence of the signal) between the first focus detection signal andthe second focus detection signal. This embodiment generates a firstfocus detection signal and a second focus detection signal based on thecorrected first viewpoint image and the corrected second viewpoint imagefor which the shading (light amount) is corrected for each of RGB, andthe captured image. Hence, this embodiment can improve the correlation(coincidence of the signal) between the first focus detection signal andthe second focus detection signal, and highly accurately detect theimage shift amount.

Where a lens is driven to an in-focus position in accordance with adefocus amount detected by the automatic focus detection, the imageshift amount distribution M_(DIS)(j, i) is multiplied by a conversioncoefficient K from an image shift amount depending on lens informationsuch as an F-number F in an image pickup lens (imaging optical system)and an exit pupil distance, to a defocus amount. Thereby, the defocusdistribution M_(Def)(j, i) can be detected. The calculation may beperformed for each image height position in the focus detecting region.

This embodiment can generate a plurality of viewpoint images with goodqualities. This embodiment can improve the detection accuracy of theimage shift amount by using a plurality of viewpoint images with goodqualities.

Fifth Embodiment

Next follows a description of a fifth embodiment according to thepresent invention. This embodiment discusses an illustrativequadrisection where Nx=2, Ny=2, and N_(LF)=4. In this embodiment, allsignals from quadrisected subpixels 201 to 204 (Nx×Ny divided firstsubpixel to N_(LF)-th subpixel) are combined for each pixel based on theinput image (LF data) corresponding to the pixel arrangement illustratedin FIG. 12 and the expression (2). The captured image is generated asthe RGB signal in the Bayer array with a resolution of the pixel numberN (=horizontal pixel number×N_(H) vertical pixel number N_(v)).

This embodiment discusses an illustrative quadrisection where Nx=2,Ny=2, N_(LF)=4, and k=1 to 3. A signal of the subpixel 201 (firstsubpixel) is selected from the quadrisected subpixels 201 to 204 arecombined for each pixel based on the LF data (input image) correspondingto the pixel arrangement illustrated in FIG. 12 and the expression (3).Then, this embodiment generates the first viewpoint image I₁(j, i) asthe RGB signal in the Bayer array with a resolution of the pixel numberN corresponding to the partial pixel area 501 in the imaging opticalsystem. This embodiment selects a signal of the subpixel 202 (secondsubpixel) from the quadrisected subpixels 201 to 204 for each pixelbased on the LF data and the expression (3). This embodiment generatesthe second viewpoint image I₂(j, i) as the RGB signal in the Bayer arraywith a resolution of the pixel number N corresponding to the partialpixel area 502 in the imaging optical system. This embodiment selects asignal of the subpixel 203 (third subpixel) from the quadrisectedsubpixels 201 to 204 for each pixel based on the LF data and theexpression (3). This embodiment generates the third viewpoint imageI₃(j, i) as the RGB signal in the Bayer array with a resolution of thepixel number N corresponding to the partial pixel area 503 in theimaging optical system.

This embodiment generates a captured image depending on a pupil regioninto which different partial pupil regions are combined, based on aninput image (LF data) acquired by an image pickup element including aplurality of arranged pixels with a plurality of photoelectricconverters configured to receive light beams that have passed differentpartial pupil regions in the imaging optical system. Then, thisembodiment generates a plurality of viewpoint images for each of thedifferent partial pupil regions. In order to generate each viewpointimage with a good quality, this embodiment performs image processing,such as a flaw correction and a shading correction, similar to the thirdembodiment, for the first viewpoint image to the fourth viewpoint image(first viewpoint image to N_(LF)-th viewpoint image) based on thecaptured image, and generates an output image.

In the step S1 in FIG. 17, this embodiment corrects shading (or a lightamount) for each of RGB in the first viewpoint image I₁ to the thirdviewpoint image I₃ (k-th viewpoint image I_(k): k=1 to N_(LF)−1) bysetting the captured image I(j, i) to a base or reference image. Thisembodiment discusses an illustrative quadrisection where Nx=2, Ny=2,N_(LF)=4, and k=1 to 3.

Initially, in the step S1, this embodiment corrects shading (lightamount) in the x direction for the k-th viewpoint image I_(k) (k=1 toN_(LF)−1) by the expressions (4A) to (7D). Next, in the expressions (4A)to (7D), the x direction is replaced with the y direction, and theshading (light amount) correction process is performed in the ydirection, and the k-th viewpoint (first corrected) image M₁I_(k) (k=1to N_(LF)−1) is generated. In performing the two stages of the shading(light amount) correction in the x direction and the shading correctionin the y direction, the pupil dividing number N_(LF) is more than thenecessary number by one so as to normalize the expressions (5A) to (5D).Hence, in the second shading correction in the y direction, themultiplication of the pupil dividing number N_(LF) for the normalizationis omitted in the expressions (5A) to (5D).

Due to the expressions (8A) to (10), the procedure of this embodiment issimilar to that in the third embodiment until the k-th viewpoint (fourthcorrected) image M₄I_(k) (k=1 to N_(LF)−1) is generated. In the step S4in FIG. 18, the N_(LF)-th viewpoint image I_(NLF)(j, i) is generated bythe following expression (16) based on the corrected captured imageMI(j, i) and the k-th viewpoint (fourth corrected) image M₄I_(k) (k=1 toN_(LF)−1). This embodiment discusses an illustrative quadrisection whereNx=2, Ny=2, and N_(LF)=4.

$\begin{matrix}{{I_{N_{LF}}\left( {j,i} \right)} = {{{MI}\left( {j,i} \right)} - {\overset{N_{LF} - 1}{\sum\limits_{k = 1}}{M_{4}{{I_{k}\left( {j,i} \right)}.}}}}} & (16)\end{matrix}$

The step S5 and subsequent steps in FIG. 18 are similar to those in thethird embodiment.

This embodiment can generate a viewpoint image with a good quality. Inthe photoelectric converter in each pixel in the image pickup element,another embodiment may increase the number of divisions, such as ninedivisions where Nx=3, Ny=3, and N_(LF)=9 and sixteen divisions whereNx=4, Ny=4, and N_(LF)=16.

As described above, the image processing apparatus (image processingcircuit 125) in each embodiment includes an acquisition unit 125 a andan image processing unit 125 b (correction unit). The acquisition unit125 a acquires a parallax image generated based on a signal of one of aplurality of photoelectric converters which receive light beams passingthrough partial pupil regions in an imaging optical system which aredifferent from each other, and acquires a captured image by combiningsignals from the plurality of photoelectric converters. The imageprocessing unit 125 b performs a correction process so as to reduce adefect (such as a dot flaw and a line flaw) contained in the parallaximage based on the captured image.

Preferably, the image processing unit 125 b corrects a pixel value(pixel signal) of the parallax image at a first position (position to becorrected) which is determined to be defective by using a pixel value(pixel signal) of the captured image at the first position. Morepreferably, the image processing unit 125 b corrects the pixel value ofthe parallax image at the first position based on a pixel value I of thecaptured image at the first position, a pixel value of the parallaximage at a second position which is not determined to be defective, anda pixel value of the captured image at the second position. The secondposition is a position of a pixel near (or around) the first position(position to be corrected). More preferably, the second position is aposition adjacent to the first position in a predetermined direction(vertical direction, horizontal direction, or oblique direction on thepixel arrangement).

Preferably, when the pixel value of the parallax image or the capturedimage at the second position is lower than a predetermined luminancevalue (parameter A0 or I0), the image processing unit 125 b replaces thepixel value with the predetermined luminance value. More preferably, thepredetermined luminance value is set to change depending on the numberof partial pupil regions. Preferably, the predetermined luminance valueis set to change depending on the first position (position to becorrected). Preferably, the predetermined luminance value is set tochange depending on imaging condition information. The imaging conditioninformation includes at least one of an ISO speed, an F-number of theimaging optical system, and an exit pupil distance.

Preferably, the image processing apparatus includes a memory (memory134) which stores defect information on the first position, or aninspector configured to inspect the defect information on the firstposition. The image processing unit 125 b performs the correctionprocess based on the defect information stored in the memory or thedefect information obtained as an inspection result by the inspector.

Preferably, the parallax image is generated by collecting lightreceiving signals from a plurality of subpixels (a plurality of firstsubpixels, a plurality of second subpixels, a plurality of thirdsubpixels, and a plurality of fourth subpixels) included in onephotoelectric converter for each of the partial pupil regions in theimaging optical system different from each other. The captured image isgenerated by collecting light receiving signals from all subpixels (aplurality of first subpixels and a plurality of second subpixels, and inaddition, a plurality of third subpixels and a plurality of fourthsubpixels if necessary) included in the plurality of photoelectricconverters.

Preferably, the parallax images include a first parallax image and asecond parallax image which correspond to respective light beams passingthrough the partial pupil regions in the imaging optical systemdifferent from each other. Then, the acquisition unit 125 a acquires thecaptured image and the first parallax image in the parallax images froman image pickup element 107 including the plurality of photoelectricconverters. The image processing unit 125 b performs the correctionprocess for the first parallax image in the parallax images. Then, theimage processing unit 125 b generates the second parallax image bysubtracting the corrected first parallax image from the captured image.Preferably, the image processing unit 125 b (refocus unit) performsrefocus process for the captured image based on the corrected parallaximage.

Preferably, the image processing unit corrects a light amount or shadingin the parallax image based on the captured image. More preferably, theimage processing unit performs a light amount correction process of aparallax image for each color in the parallax image based on thecaptured image. More preferably, the image processing unit performs thelight amount correction process for the parallax image based on theprojection signal of the captured image and the projection signal of theparallax image. More preferably, after the image processing unitperforms a light amount correction process for a parallax image, theimage processing unit corrects a parallax image so as to reduce thedefect contained in the parallax image after the light amount correctionprocess is performed based on the captured image.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

Each embodiment can provide an image processing apparatus, an imagepickup apparatus, an image pickup apparatus, an image processing method,and a non-transitory computer-readable storage medium, each of which cangenerate a parallax image having an improved quality.

While the present invention has been described with reference toexemplary embodiments, the invention is not limited to the disclosedexemplary embodiments and various variations and modifications may bemade without departing from the scope of the invention.

1-23. (canceled)
 24. An image processing apparatus comprising: (A) amemory storing instructions; (B) one or more processors that execute theinstructions stored in the memory; and (C) an image processing circuitthat, based on the instructions executed by the one or more processors,is configured to function as: (a) an acquisition unit configured toacquire (i) a parallax image generated based on a signal from one of aplurality of photoelectric converters corresponding to a same microlensfrom an image pickup element, wherein the image pickup element includesa plurality of pixels that includes the plurality of photoelectricconverters, and wherein each of the plurality of pixels corresponds to arespective one of a plurality of microlenses, and (ii) a composite imagegenerated by combining a plurality of signals from the plurality ofphotoelectric converters; (b) an image processing unit configured (i) tocalculate a correction value by using pixel values of the compositeimage corresponding to a defect, and (ii) to correct one or more pixelvalues of the parallax image that include the defect with thecorresponding correction value; and (c) a storage unit configured tostore in a storage medium the parallax image corrected by the imageprocessing unit.
 25. The image processing apparatus according to claim24, further comprising a determination unit configured to determinewhether the parallax image contains the defect.
 26. The image processingapparatus according to claim 25, wherein the image processing unitcorrects the pixel value of the parallax image at a first position thatcontains the defect, based on a comparison of the pixel value of theparallax image with a pixel value of the composite image at the firstposition.
 27. The image processing apparatus according to claim 26,wherein the image processing unit corrects the pixel value of theparallax image at the first position based on (1) the pixel value of thecomposite image at the first position, (2) a pixel value of the parallaximage at a second position that does not contain a defect, and (3) apixel value of the composite image at the second position.
 28. The imageprocessing apparatus according to claim 27, wherein the second positionis a position adjacent to the first position in a predetermineddirection.
 29. The image processing apparatus according to claim 27,wherein when a luminance value of one of the pixel value of the parallaximage at the second position and the pixel value of the composite imageat the second position is less than a predetermined luminance value, theimage processing unit replaces the pixel value of the parallax imagewith the predetermined luminance value.
 30. The image processingapparatus according to claim 29, wherein the predetermined luminancevalue is set to change depending on a number of partial pupil regions.31. The image processing apparatus according to claim 29, wherein thepredetermined luminance value is set to change depending on the firstposition.
 32. The image processing apparatus according to claim 29,wherein the predetermined luminance value is set to change depending onimaging condition information.
 33. The image processing apparatusaccording to claim 32, wherein the imaging condition informationincludes at least one of (1) an International Organization ofStandardization (ISO) speed, (2) an F-number of an imaging opticalsystem which forms an optical image on the image pickup element, and (3)an exit pupil distance.
 34. The image processing apparatus according toclaim 26, wherein the memory is configured to store defect informationon the first position, and wherein the image processing unit correctsthe parallax image based on the stored defect information.
 35. The imageprocessing apparatus according to claim 34, further comprising aninspector configured to inspect defect information on the firstposition, wherein the image processing unit corrects the parallax imagebased on the defect information.
 36. The image processing apparatusaccording to claim 24, wherein the parallax image is generated bycollecting light receiving signals from a plurality of subpixelsincluded in the one photoelectric converter, of the plurality ofphotoelectric converters, for each of the partial pupil regions in animaging optical system which forms an optical image on the image pickupelement, the partial pupil regions being different from each other, andwherein the composite image is generated by collecting light receivingsignals from all subpixels included in the plurality of photoelectricconverters.
 37. The image processing apparatus according to claim 24,wherein, where a plurality of parallax images includes a first parallaximage and a second parallax image, the first parallax image and thesecond parallax image corresponding to light beams passing through thepartial pupil regions in an imaging optical system which forms anoptical image on the image pickup element, the partial pupil regionsbeing different from each other, the acquisition unit acquires thecomposite image and the first parallax image, among the plurality ofparallax images, from an image pickup element including the plurality ofphotoelectric converters, and wherein the image processing unit isfurther configured: to correct the first parallax image, among theplurality of parallax images; and to generate the second parallax imageby subtracting the corrected first parallax image from the capturedimage.
 38. The image processing apparatus according to claim 24, whereinthe image processing unit performs a refocus process for the compositeimage based on the corrected parallax image.
 39. The image processingapparatus according to claim 24, wherein the image processing unitperforms a light amount correction process for the parallax image basedon the composite image.
 40. The image processing apparatus according toclaim 39, wherein the image processing unit performs the light amountcorrection process for the parallax image based on the composite imagefor each color of the parallax image based on the composite image. 41.The image processing apparatus according to claim 39, wherein the imageprocessing unit performs the light amount correction process for theparallax image based on a projection signal of the composite image and aprojection signal of the parallax image.
 42. The image processingapparatus according to claim 39, wherein the image processing unitcorrects the parallax image after the light amount correction process isperformed, based on the composite image, after the light amountcorrection process is performed for the parallax image.
 43. An imagepickup apparatus comprising: (A) an image pickup element including aplurality of arrayed pixels that include a plurality of photoelectricconverters that receive light beams passing through partial pupilregions in an imaging optical system which forms an optical image on theimage pickup element, the partial pupil regions being different fromeach other; (B) a memory storing instructions; (C) one or moreprocessors that execute the instructions stored in the memory; and (D)an image processing circuit that, based on the instructions executed bythe one or more processors, is configured to function as: (a) anacquisition unit configured to acquire (i) a parallax image generatedbased on a signal from one of the plurality of photoelectric converterscorresponding to a same microlens from the image pickup element, whereinthe image pickup element includes a plurality of pixels that includesthe plurality of photoelectric converters, and wherein each of theplurality of pixels corresponds to a respective one of a plurality ofmicrolenses, and (ii) a composite image generated by combining signalsfrom the plurality of photoelectric converters; (b) an image processingunit configured (i) to calculate a correction value by using pixelvalues of the composite image corresponding to a defect, and (ii) tocorrect one or more pixel values of the parallax image that include thedefect with the corresponding correction value; and (c) a storage unitconfigured to store in a storage medium the parallax image corrected bythe image processing unit.
 44. The image pickup apparatus according toclaim 43, wherein the image pickup element includes the plurality ofphotoelectric converters for a single microlens, and the microlens istwo-dimensionally arrayed.
 45. An image processing method comprising thesteps of: acquiring (i) a parallax image generated based on a signalfrom one of a plurality of photoelectric converters corresponding to asame microlens from an image pickup element, wherein the image pickupelement includes a plurality of pixels that includes the plurality ofphotoelectric converters, and wherein each of the plurality of pixelscorresponds to a respective one of a plurality of microlenses, and (ii)a composite image generated by combining a plurality of signals from theplurality of photoelectric converters; calculating a correction value byusing signals pixel values of the composite image corresponding to adefect; correcting one or more pixel values of the parallax image thatinclude the defect with the corresponding correction value; and storingin a storage medium the parallax image corrected by the correcting. 46.A non-transitory computer-readable storage medium storing a program thatcauses a computer to execute an image processing method comprising thesteps of: acquiring (i) a parallax image generated based on a signalfrom one of a plurality of photoelectric converters corresponding to asame microlens from an image pickup element, wherein the image pickupelement includes a plurality of pixels that includes the plurality ofphotoelectric converters, and wherein each of the plurality of pixelscorresponds to a respective one of a plurality of microlenses, and (ii)a composite image generated by combining a plurality of signals from theplurality of photoelectric converters; calculating a correction value byusing the pixel values of the composite image corresponding to a defect;correcting one or more pixel values of the parallax image with thecorresponding correction value; and storing in a storage medium theparallax image corrected by correcting.
 47. The image processingapparatus according to claim 24, wherein the correction value Ac(i, j),for a pixel at a position (i, j) in the parallax image, is calculatedusing a pixel value A(i, j) of the parallax image and a pixel value I(i,j) of the composite image, in the following expression:${{Ac}\left( {j,i} \right)} = {\frac{\max \left( {\sum_{k = {i - 1}}^{i + 1}\left\lbrack {{{A\left( {{j - 1},k} \right)} + {A\left( {{j + 1},k} \right)}},{A0}} \right\rbrack} \right)}{\max \left( {\sum_{k = {i - 1}}^{i + 1}\left\lbrack {{{I\left( {{j - 1},k} \right)} + {I\left( {{j + 1},k} \right)}},{I0}} \right\rbrack} \right)} \times {I\left( {j,i} \right)}}$where A0 and I0 are values used to stabilize a calculation value and tosuppress an amplification of a noise.